Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........................X........................] duration: 6.002206621s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:11:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.002234335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...........................X.......................] duration: 5.964766979s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:11:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.964796344s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 346.170166ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6442c155ed5dd444 FAIL (2025-11-14 04:10:50 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:59564->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 346.1924ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........................X......................] duration: 6.018970173s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:10:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.019101052s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 423.233851ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6442c155ed5dd444 FAIL (2025-11-14 04:10:17 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:59564->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 423.258459ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........................X.....................] duration: 5.907518257s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:09:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.90754631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........................X....................] duration: 5.914774s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:09:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.914807945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...........................X...................] duration: 5.967622136s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:08:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.967739128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...........................X..................] duration: 5.969727727s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:07:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.969759296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...........................X.................] duration: 6.044976557s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:06:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.045021233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...........................X................] duration: 5.964022445s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:06:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.964054507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...........................X...............] duration: 6.150091131s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.15012243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...........................X..............] duration: 5.986715908s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:04:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.986746255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...........................X.............] duration: 5.879795823s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:04:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.879828856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...........................X............] duration: 6.081154705s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:03:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.081267682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...........................X...........] duration: 5.93778882s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:02:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.937818175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...........................X..........] duration: 5.88612274s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:01:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.886152696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...........................X.........] duration: 5.978983667s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 04:00:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.97901601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...........................X........] duration: 5.933292379s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:59:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.933315334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...........................X.......] duration: 6.039757429s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:58:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.039787416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...........................X......] duration: 5.934537158s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:57:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.934570182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...........................X.....] duration: 5.972744934s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:56:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.972773909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...........................X....] duration: 5.929611543s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.929632073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X...........................X...] duration: 6.044909151s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:55:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.04493513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...........................X..] duration: 5.893757387s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:54:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.893901382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........................X.] duration: 5.962351084s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:53:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.962372595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........................X] duration: 30.331057834s trace: https://jaeger.dev.storj.io/trace/1044b29d27a1bc30 FAIL (2025-11-14 03:53:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 3219999 (115h48m30.61469046s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063325610 (115h48m30.614796071s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.063895234 (115h48m30.615365695s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064350882 (115h48m30.615821343s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.064784418 (115h48m30.616254869s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065308686 (115h48m30.616779147s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.065766728 (115h48m30.617237189s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066315213 (115h48m30.617785674s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.066926716 (115h48m30.618397177s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.067469991 (115h48m30.618940462s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.068967974 (115h48m30.620438435s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.069199745 (115h48m30.620670206s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.070997598 (115h48m30.622468059s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.071082831 (115h48m30.622553282s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.072733424 (115h48m30.624203895s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:32.074291691 (115h48m30.625762142s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Uploading piece... Nov 14 03:52:33.785296023 (115h48m32.336766484s) [upload=101876]: Next returned nil segment Nov 14 03:52:33.785314808 (115h48m32.336785269s) [upload=101876]: Waiting for error group managing segments... Nov 14 03:52:34.115404913 (115h48m32.666875374s) [noise=true , address=45.91.116.13:4155 , piece_id=3B7JDU2IJHHOYEJHSHW2UKXTHFB76DKJG3KIM5H6PC3T6KBKIPJQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.169217042 (115h48m32.720687503s) [noise=true , address=45.91.116.6:4975 , piece_id=WQXOW4DDKXUEAJOVHLVCGP7YISCSIAN2PEN4NATW7NVCP2VGBKNQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.176024124 (115h48m32.727494585s) [noise=true , address=77.90.31.11:4555 , piece_id=HTZZEQCFRJ6VGJA6ABVJBHSMSNZQPBJT5JJMCUFCJ764KAW43DHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.182050910 (115h48m32.733521371s) [noise=true , address=45.91.116.12:4225 , piece_id=J4AS7LXOZUR3B5YUPB2ARCRXXNP64PYVQU7APL43OHUH4CFKSDVA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.193028663 (115h48m32.744499124s) [noise=true , address=45.91.116.11:4119 , piece_id=VMPTX5JZ3DH425R6KNGOE7FMEBF7FZ5QIH6DJZLQOCLVPH4OA5FQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.224900795 (115h48m32.776371256s) [noise=true , address=45.91.116.12:4227 , piece_id=XTY4DRRQWVLFSDBFGGFUZJANDBHTOB56YSYYWOZCR7OSGSOVO3XQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:52:34.239033406 (115h48m32.790503867s) [noise=true , address=45.91.116.14:4189 , piece_id=32JHLVTW3RNO3HEUJATDBHTVSFNJSOHZGCQOV6A26WB2Q4UXGVFQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: <nil> Nov 14 03:53:01.094210755 (115h48m59.645681256s) [noise=true , address=145.79.227.11:4729 , piece_id=N5L7HI5PBBAMP7EBUNKEJ2SIML5PYT6U4V7QCHH53PLU2BKHSBSQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094342466 (115h48m59.645812937s) [noise=true , address=145.79.227.10:4401 , piece_id=OQS366USBNZC5LEZMH3PZRBS7WBKJLDBLDDGKQVRXU743MR6TRRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47410->145.79.227.10:4401: use of closed network connection Nov 14 03:53:01.094502721 (115h48m59.645973192s) [noise=true , address=145.79.227.32:4836 , piece_id=NUM3QPXST6ERVSWS6PJ3YP2WA4XOO4T62JGWY4SVQCWVPPPOZ3WA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46418->145.79.227.32:4836: use of closed network connection Nov 14 03:53:01.094527148 (115h48m59.645997609s) [noise=true , address=145.79.227.28:4727 , piece_id=GMYRMPYA7VUTB5OSDPDOZLUJEV75H3MEEQ6BNJUPOTUEEABHOZHQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: rpc: dial tcp 145.79.227.28:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094672995 (115h48m59.646143456s) [noise=true , address=145.79.227.21:4475 , piece_id=S2Y4GBELVTMFHPGRJAXBXZCJNHXWNEWYB4KHNZGZMWJHASKZ2GIQ, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: read tcp 188.166.225.170:34888->145.79.227.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.094966164 (115h48m59.646436625s) [noise=true , address=145.79.227.23:4547 , piece_id=J2WWVBPVIXV3FTROKFN4FREPRD456AS43Q236DP7MO7D5245PKRA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:37164->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095198325 (115h48m59.646668786s) [noise=true , address=145.79.227.6:4259 , piece_id=7K4ELW6OB6EN24FC6IWPSGODJE7Z3UUPFB4KSUW4MU2HNL6SNW5A, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:43758->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095418525 (115h48m59.646888986s) [noise=true , address=145.79.227.17:4970 , piece_id=IQRKTXVGWHFZ7ECA4AAIWVDSUQYVBWKOHDRGTD7HKRLKIVUICRHA, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:58470->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095682127 (115h48m59.647152588s) [noise=true , address=145.79.227.20:4438 , piece_id=5SVOJNRNUN5AR6DU6X72TSWGCGZE3NUNTFFXQMSECR3RCWWUVC6Q, seg_pos={0 0}, upload=101876]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:33456->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:53:01.095719017 (115h48m59.647189478s) [seg_pos={0 0}, upload=101876]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 14 03:53:01.421732930 (115h48m59.973203391s) [upload=101876]: Done uploading segments... Nov 14 03:53:01.421782805 (115h48m59.973253266s) [upload=101876]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.331081279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.941402695s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.941433243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.063033188s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:50:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.063061463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.06026614s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:50:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.06029796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.971945182s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:48:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.971979367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.913805006s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:48:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.913822108s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.96359626s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:47:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.963623622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.934297169s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:46:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.934330213s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 45.316142ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:46:00 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 45.343514ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.888904259s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:45:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.888951981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.067253824s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:45:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.067291184s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 68.714456ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:45:02 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.746606ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 57.850318ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:44:41 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 57.877951ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.948851856s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:44:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.9488858s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 65.493713ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:44:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 65.521074ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 53.77624ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:43:26 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.806729ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.917910564s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:43:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.917931182s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 58.949151ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:42:38 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.989037ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.923252926s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.923287181s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 331.828788ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/590d6bce2539f822 FAIL (2025-11-14 03:42:26 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:35086->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 331.893721ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 51.139389ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:42:06 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 51.187161ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.066777377s trace: https://jaeger.dev.storj.io/trace/57998300e01e0c2f FAIL (2025-11-14 03:41:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== ss=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.884971322 (115h26m33.436441793s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.885891113 (115h26m33.437361564s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.887874872 (115h26m33.439345333s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.889455992 (115h26m33.440926493s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890058981 (115h26m33.441529442s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890165263 (115h26m33.441635714s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890478780 (115h26m33.441949241s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890782909 (115h26m33.442253371s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.890936381 (115h26m33.442406842s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891341754 (115h26m33.442812225s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891511727 (115h26m33.442982188s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.891779026 (115h26m33.443249477s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:34.892006599 (115h26m33.44347705s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Uploading piece... Nov 14 03:30:36.566889712 (115h26m35.118360163s) [upload=101556]: Next returned nil segment Nov 14 03:30:36.566911953 (115h26m35.118382404s) [upload=101556]: Waiting for error group managing segments... Nov 14 03:30:36.964404817 (115h26m35.515875278s) [noise=true , address=45.91.116.14:4190 , piece_id=DDLNFXEG2TQDBANM7FU4FI4XWAJNST5OGHWW76SO5EW3ZSP6BUJA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:36.996835986 (115h26m35.548306447s) [noise=true , address=77.90.31.15:4698 , piece_id=SLR3LK3AKC4QZY5ZJZUMGN5E5ADCRKRMPKIRRTZNYDRZDAZFPNLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.021055808 (115h26m35.572526269s) [noise=true , address=77.90.31.17:4772 , piece_id=U273A5D3UX3Z2BRVXHR3ABF4IFDAQIAALKNSU24RRJZNWTBWEVRA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.022512001 (115h26m35.573982462s) [noise=true , address=45.91.116.4:4937 , piece_id=35UICEJ5URRTMXKOGG3CGCZDDETJV3YPLNA2TEGARDDBDXQ4EZBA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.030221303 (115h26m35.581691764s) [noise=true , address=45.91.116.7:4011 , piece_id=TXJEP44YOLSDTQ752NIJDN2VAVUSVIE6ATRBMCGPPNCUNODBKHCA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:30:37.059091300 (115h26m35.610561761s) [noise=true , address=77.90.31.21:4879 , piece_id=UACCKFTCKBQG7I3G6MK77LVA4GKGGPLPWX2ZTAJI6J7ROUPWBVIA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: <nil> Nov 14 03:31:03.921849669 (115h27m2.47332013s) [noise=true , address=145.79.227.31:4797 , piece_id=AEPWX7YKOE5VWBFR5LZCJB4OPWW4MCACTNN5O43WQZXIDO7CX2HQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55326->145.79.227.31:4797: use of closed network connection Nov 14 03:31:03.921977072 (115h27m2.473447533s) [noise=true , address=145.79.227.17:4972 , piece_id=2BMIL4EXKVR2W2W63RK6MJHLGWEEOJ74ND6D2NKOFXM4L4G2D6GA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57642->145.79.227.17:4972: use of closed network connection Nov 14 03:31:03.922075539 (115h27m2.47354599s) [noise=true , address=145.79.227.16:4910 , piece_id=ASET5RFPUTNK2DJADM2SWQODMEDNDZTIFKWQX5NTEGMEI433RBGQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33048->145.79.227.16:4910: use of closed network connection Nov 14 03:31:03.922220054 (115h27m2.473690515s) [noise=true , address=145.79.227.9:4367 , piece_id=EG57M62DUIPSS3ZTBKA25MJWYJHXGYOVE2UIU4FIRIT2GHBSLPAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:33770->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922719866 (115h27m2.474190327s) [noise=true , address=145.79.227.6:4258 , piece_id=JMEWZPODLXAHX7RYJBNG2DMFPKJ5OVNVZF5MT5UH2ISJYIRR7Y6Q, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: read tcp 188.166.225.170:48224->145.79.227.6:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.922998376 (115h27m2.474468837s) [noise=true , address=145.79.227.8:4330 , piece_id=OTTXKYUKF4KBGR6CSQQZ3J34SHJJ7VFQR4UFQSCXTOP64XRTSRSQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:35520->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923210620 (115h27m2.474681081s) [noise=true , address=145.79.227.25:4619 , piece_id=JU5CBH2BMZNQTGAMRMGWAVPOQ7FGS5LJSDKIA6PHZTM6X6BXS43A, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:34932->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923255957 (115h27m2.474726418s) [noise=true , address=145.79.227.21:4474 , piece_id=KNQ2Z4BISMM5WRNWUFIM6FISKC4PWXXWLN23WEG3W3LXZX5I5NAA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: read tcp 188.166.225.170:49270->145.79.227.21:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923569434 (115h27m2.475039895s) [noise=true , address=145.79.227.2:4113 , piece_id=K3SNAFJKQVBA5VU5CSNJ5THQ7KYNQ5L2QGWUOK6TSN6OP3XYLUVA, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: read tcp 188.166.225.170:39106->145.79.227.2:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923652051 (115h27m2.475122512s) [noise=true , address=145.79.227.20:4437 , piece_id=IGKYDKJIYMWMENQ65RP7CIVN2TYUJUGC3F5X7JZSFFJSCXUA5TLQ, seg_pos={0 0}, upload=101556]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:45728->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 03:31:03.923686977 (115h27m2.475157438s) [seg_pos={0 0}, upload=101556]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 03:31:04.172610117 (115h27m2.724080568s) [upload=101556]: Done uploading segments... Nov 14 03:31:04.172665181 (115h27m2.724135643s) [upload=101556]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.066801011s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 59.420328ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/493eac2491a63607 FAIL (2025-11-14 03:41:28 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:48896->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 59.51594ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 377.176279ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/590d6bce2539f822 FAIL (2025-11-14 03:41:23 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:35086->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 377.209001ms core website

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)