Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.724778762s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.72480357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.696173832s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:42:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.696221091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.86360147s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:41:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.863623231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.16005367s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:40:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.160087775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.636948709s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:40:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.636978394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.794347766s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:39:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.794367714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.816831146s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:38:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.816874568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.881522603s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.881554704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.762678662s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:36:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.762709893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.78939345s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:36:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.78942563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.865884564s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:35:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.865915342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.03682434s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:35:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.036850991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.868922487s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:34:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.868967953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.77089682s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:33:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.770923649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.93379494s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:32:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.933826992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.964206033s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:32:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.964234197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.664062717s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.66409598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.79832142s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:30:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.798354934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.228789922s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:30:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.228817554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.730887241s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.730910524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.763713312s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:28:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.763742096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.658144758s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:27:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.658173253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.690469881s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:26:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.690495218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.159193206s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:25:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.159219837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.63636285s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:24:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.636395023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.934803382s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:23:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.934838659s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 14.270158853s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:22:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.270186615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.691721823s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:22:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.69175172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.685016937s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:21:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.685042336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.855391821s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:20:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.855421346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.852659027s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.852733178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 13.666298177s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:18:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 13.666314148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.249689893s trace: https://jaeger.dev.storj.io/trace/7d7482285de47d39 FAIL (2025-11-26 04:18:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ... Nov 26 04:18:02.078866398 (404h14m0.630336859s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079056660 (404h14m0.630527121s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079363073 (404h14m0.630833534s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079767873 (404h14m0.631238335s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.079801347 (404h14m0.631271828s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080268257 (404h14m0.631738728s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080385139 (404h14m0.6318556s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.080904648 (404h14m0.63237512s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081245638 (404h14m0.632716109s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081352391 (404h14m0.632822852s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081840841 (404h14m0.633311302s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.081865388 (404h14m0.633335849s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.082498303 (404h14m0.633968774s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:02.084982413 (404h14m0.636452884s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Uploading piece... Nov 26 04:18:03.707821995 (404h14m2.259292466s) [upload=355969]: Next returned nil segment Nov 26 04:18:03.707839479 (404h14m2.25930994s) [upload=355969]: Waiting for error group managing segments... Nov 26 04:18:04.097259042 (404h14m2.648729563s) [noise=true , address=145.79.227.17:4969 , piece_id=CNHAEZFOOM4ZT74N2667DBFZDZ522IMY7YA6GK5GSVMWXQQTVMRA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.123197744 (404h14m2.674668205s) [noise=true , address=145.79.227.24:4581 , piece_id=WK3CWWGI7JTVV2B22SQA4OT75K6EDSM36MTHZOBVXF4IEDTZRRPA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.126869185 (404h14m2.678339646s) [noise=true , address=145.79.227.27:4689 , piece_id=WPJ4WRKE3VL6MYGHF4MKSQ3EQY2WPHM35V4J3VCMYVPZKPX6YQ2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.150032532 (404h14m2.701502993s) [noise=true , address=145.79.227.21:4474 , piece_id=P4XSQ2R7O5U4BA253IFU3WLA2HINBOVACUSD2Q4Q4I3U3TOW764Q, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.199303663 (404h14m2.750774124s) [noise=true , address=77.90.31.11:4554 , piece_id=ZBJTQPN6STEJJ5KWI35IVAVTDLZ7H2ZJP66CNG2Z33NFN7XOMD2A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.234808390 (404h14m2.786278851s) [noise=true , address=77.90.31.10:4519 , piece_id=3WRHXLCQR7KY4UCR7GREP7C6H4NWW2ZXIX23VBHFJGDDEW4JYMZQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:04.282578253 (404h14m2.834048714s) [noise=true , address=45.91.116.14:4190 , piece_id=MWG3LAK3H2B4CZMBOWS6PTPZVS366HGFWCY3GDLOMWDWB2F4AGSQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:05.347192587 (404h14m3.898663048s) [noise=true , address=145.79.227.42:4122 , piece_id=UL4AXMPV5GA4KX2BXR6NJXUX65VGHXG2O34O4DD5N4QJNMGGMNUA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: <nil> Nov 26 04:18:31.131246400 (404h14m29.682716861s) [noise=true , address=145.79.227.40:4049 , piece_id=CB6SVEGVGWRVFN4ZUKI2GJWXMHEY4GBJ4R2SOBBT3YUY5CX6FA6A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48644->145.79.227.40:4049: use of closed network connection Nov 26 04:18:31.131401364 (404h14m29.682871825s) [noise=true , address=145.79.227.20:4438 , piece_id=257KBABEFA7UTSQX4RBSMVCA4FJSHYD7NU4KXOHIUNQ2EY5VJL5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43368->145.79.227.20:4438: use of closed network connection Nov 26 04:18:31.131459455 (404h14m29.682929926s) [noise=true , address=145.79.227.32:4835 , piece_id=32AUI64HDLPXOVK535CISVAT2CQYGDC74DUBMXET2RI3SSLSMFOA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55720->145.79.227.32:4835: use of closed network connection Nov 26 04:18:31.131523117 (404h14m29.682993578s) [noise=true , address=145.79.227.6:4257 , piece_id=5V6OAASB4YY7ZOHSDYOIFEWYUQWKY4FF7SGBXYLS27KYQ4WWNXMA, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43370->145.79.227.6:4257: use of closed network connection Nov 26 04:18:31.131657422 (404h14m29.683127884s) [noise=true , address=145.79.227.10:4401 , piece_id=CWL7DD677S3SAU3EPX7E5W2WKXZWUT2BZP6COVJYIYX5WY3ZRZ5A, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55120->145.79.227.10:4401: use of closed network connection Nov 26 04:18:31.132100355 (404h14m29.683570816s) [noise=true , address=145.79.227.33:4869 , piece_id=2SR3O2I2PIYBLWC5YTENETKLHJKFNULEAOH62UXTPDGTPTXCNTPQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38902->145.79.227.33:4869: use of closed network connection Nov 26 04:18:31.132214393 (404h14m29.683684844s) [noise=true , address=145.79.227.14:4840 , piece_id=53DMEU2EKZHWUMWLDHMZM4VZA3DQ357ACWEECEZLH43TIYGFTNHQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33032->145.79.227.14:4840: use of closed network connection Nov 26 04:18:31.132312088 (404h14m29.683782579s) [noise=true , address=145.79.227.41:4086 , piece_id=NI7NNBF2D7SC6KYKCWZMKGRGDMJ6KD6L4DUDT2TAMAJILZ35GXOQ, seg_pos={0 0}, upload=355969]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: expected piece hash; context deadline exceeded; EOF Nov 26 04:18:31.132352997 (404h14m29.683823458s) [seg_pos={0 0}, upload=355969]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 26 04:18:31.379208213 (404h14m29.930678674s) [upload=355969]: Done uploading segments... Nov 26 04:18:31.379254551 (404h14m29.930725013s) [upload=355969]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.249718938s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.447924634s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447946896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.444878002s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:29:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444904744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.383869741s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:29:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383929345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.381562899s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.381688627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.491342318s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:28:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491368979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 4.318536751s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:27:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.318565295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.526497519s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526543808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.582255985s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:26:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582287884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.465698006s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:25:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466375487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.431454331s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:24:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.431482163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 4.276490817s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:23:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.276522216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.522303614s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:23:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522720058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.429392112s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:23:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429420747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.40610448s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:21:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40615086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.531415805s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:21:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531517979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.580427813s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:20:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.580459974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.584633791s trace: https://jaeger.dev.storj.io/trace/383101a444ce503f FAIL (2025-11-26 01:20:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5846648s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)