Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...X........XX........X.............................X..] duration: 5.980715396s trace: https://jaeger.dev.storj.io/trace/42cd0025356bc4b6 FAIL (2025-11-13 23:51:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, Piece ID: KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q) context deadline exceeded ======LOGS====== estItem_ObjectBegin Nov 13 23:50:20.296282661 (111h46m18.847753122s) [upload=98334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:50:20.550320931 (111h46m19.101791402s) [seg_pos={0 0}, upload=98334]: Begin upload segment... Nov 13 23:50:20.550376987 (111h46m19.101847448s) [seg_pos={0 0}, upload=98334]: Done begin upload segment. Nov 13 23:50:20.550412856 (111h46m19.101883317s) [noise=true , address=145.79.227.10:4402 , piece_id=AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.551653127 (111h46m19.103123588s) [noise=true , address=77.90.31.10:4517 , piece_id=UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.552493087 (111h46m19.103963538s) [noise=true , address=145.79.227.23:4547 , piece_id=BLY6OGAICCKMUNKPXKDMVJ3EAAKFTSNENTHKKRP7BG2UH7CCMYMQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.552926622 (111h46m19.104397094s) [noise=true , address=145.79.227.30:4762 , piece_id=WI5NJJ6PJ56HHQLO62CYNM7UGXA2ORQBK6R3GNY6O4LXD3SWLCAA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.553368324 (111h46m19.104838785s) [noise=true , address=45.91.116.12:4225 , piece_id=B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.554447308 (111h46m19.1059178s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.554916451 (111h46m19.106386902s) [noise=true , address=145.79.227.31:4797 , piece_id=S4HRWXQLYLOI6YP3WMGNOCNPJAEIPQNQNBP4XGSYUZOT4NKGV7OQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.555399110 (111h46m19.106869572s) [noise=true , address=77.90.31.15:4699 , piece_id=KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.556865883 (111h46m19.108336344s) [noise=true , address=77.90.31.12:4590 , piece_id=LZ7MU5MAP6IATUSYSC7UIDST6NEJDLXDTRIL3AG4PNSPH4LLDMUA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.558207068 (111h46m19.109677529s) [noise=true , address=45.91.116.4:4939 , piece_id=BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.559947131 (111h46m19.111417592s) [noise=true , address=145.79.227.32:4835 , piece_id=AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.560540791 (111h46m19.112011252s) [noise=true , address=145.79.227.28:4726 , piece_id=AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.562062880 (111h46m19.113533341s) [noise=true , address=145.79.227.17:4972 , piece_id=2YAJSNY7FQXP3WCOSFQAFRWHUI3N2XGREHPZ4IBTI734FF3RXQWA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.562668593 (111h46m19.114139054s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.569385884 (111h46m19.120856355s) [noise=true , address=145.79.227.24:4581 , piece_id=CSFW35TLB7SZSYXRGPENRC4LG46JK3PV4ONSKHQEGMDDZIKYNT3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.572194983 (111h46m19.123665444s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:22.292097836 (111h46m20.843568317s) [upload=98334]: Next returned nil segment Nov 13 23:50:22.292115399 (111h46m20.84358586s) [upload=98334]: Waiting for error group managing segments... Nov 13 23:50:22.631846466 (111h46m21.183316927s) [noise=true , address=77.90.31.12:4590 , piece_id=LZ7MU5MAP6IATUSYSC7UIDST6NEJDLXDTRIL3AG4PNSPH4LLDMUA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.664080115 (111h46m21.215550566s) [noise=true , address=45.91.116.4:4939 , piece_id=BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.664601498 (111h46m21.216071959s) [noise=true , address=45.91.116.12:4225 , piece_id=B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.676547116 (111h46m21.228017577s) [noise=true , address=77.90.31.15:4699 , piece_id=KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.707487042 (111h46m21.258957493s) [noise=true , address=77.90.31.10:4517 , piece_id=UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:39.037917956 (111h46m37.589388417s) [noise=true , address=145.79.227.24:4581 , piece_id=CSFW35TLB7SZSYXRGPENRC4LG46JK3PV4ONSKHQEGMDDZIKYNT3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:41.226483506 (111h46m39.777953967s) [noise=true , address=145.79.227.23:4547 , piece_id=BLY6OGAICCKMUNKPXKDMVJ3EAAKFTSNENTHKKRP7BG2UH7CCMYMQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:41.699020421 (111h46m40.250490873s) [noise=true , address=145.79.227.31:4797 , piece_id=S4HRWXQLYLOI6YP3WMGNOCNPJAEIPQNQNBP4XGSYUZOT4NKGV7OQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:43.149006879 (111h46m41.70047733s) [noise=true , address=145.79.227.32:4835 , piece_id=AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:43.431104588 (111h46m41.98257505s) [noise=true , address=145.79.227.10:4402 , piece_id=AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:44.182010887 (111h46m42.733481348s) [noise=true , address=145.79.227.30:4762 , piece_id=WI5NJJ6PJ56HHQLO62CYNM7UGXA2ORQBK6R3GNY6O4LXD3SWLCAA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:46.418512937 (111h46m44.969983398s) [noise=true , address=145.79.227.17:4972 , piece_id=2YAJSNY7FQXP3WCOSFQAFRWHUI3N2XGREHPZ4IBTI734FF3RXQWA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:48.360715564 (111h46m46.912186025s) [noise=true , address=145.79.227.28:4726 , piece_id=AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:48.360865739 (111h46m46.91233619s) [seg_pos={0 0}, upload=98334]: Segment reached optimal threshold of 13 pieces. Nov 13 23:50:48.361094996 (111h46m46.912565457s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection Nov 13 23:50:48.361103071 (111h46m46.912573533s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361417480 (111h46m46.912887941s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection Nov 13 23:50:48.361430746 (111h46m46.912901217s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361561314 (111h46m46.913031765s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection Nov 13 23:50:48.361569741 (111h46m46.913040192s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361591251 (111h46m46.913061712s) [seg_pos={0 0}, upload=98334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 13 23:50:48.361690230 (111h46m46.913160691s) [upload=98334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 23:50:48.361696562 (111h46m46.913167023s) [upload=98334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 23:50:48.674005882 (111h46m47.225476343s) [upload=98334]: Done uploading segments... Nov 13 23:50:48.674031299 (111h46m47.225501811s) [upload=98334]: Upload finished. err: <nil> Duration:* 5.980744171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...X........XX........X.............................X.] duration: 5.870463005s trace: https://jaeger.dev.storj.io/trace/42cd0025356bc4b6 FAIL (2025-11-13 23:51:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, Piece ID: KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q) context deadline exceeded ======LOGS====== estItem_ObjectBegin Nov 13 23:50:20.296282661 (111h46m18.847753122s) [upload=98334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:50:20.550320931 (111h46m19.101791402s) [seg_pos={0 0}, upload=98334]: Begin upload segment... Nov 13 23:50:20.550376987 (111h46m19.101847448s) [seg_pos={0 0}, upload=98334]: Done begin upload segment. Nov 13 23:50:20.550412856 (111h46m19.101883317s) [noise=true , address=145.79.227.10:4402 , piece_id=AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.551653127 (111h46m19.103123588s) [noise=true , address=77.90.31.10:4517 , piece_id=UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.552493087 (111h46m19.103963538s) [noise=true , address=145.79.227.23:4547 , piece_id=BLY6OGAICCKMUNKPXKDMVJ3EAAKFTSNENTHKKRP7BG2UH7CCMYMQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.552926622 (111h46m19.104397094s) [noise=true , address=145.79.227.30:4762 , piece_id=WI5NJJ6PJ56HHQLO62CYNM7UGXA2ORQBK6R3GNY6O4LXD3SWLCAA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.553368324 (111h46m19.104838785s) [noise=true , address=45.91.116.12:4225 , piece_id=B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.554447308 (111h46m19.1059178s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.554916451 (111h46m19.106386902s) [noise=true , address=145.79.227.31:4797 , piece_id=S4HRWXQLYLOI6YP3WMGNOCNPJAEIPQNQNBP4XGSYUZOT4NKGV7OQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.555399110 (111h46m19.106869572s) [noise=true , address=77.90.31.15:4699 , piece_id=KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.556865883 (111h46m19.108336344s) [noise=true , address=77.90.31.12:4590 , piece_id=LZ7MU5MAP6IATUSYSC7UIDST6NEJDLXDTRIL3AG4PNSPH4LLDMUA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.558207068 (111h46m19.109677529s) [noise=true , address=45.91.116.4:4939 , piece_id=BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.559947131 (111h46m19.111417592s) [noise=true , address=145.79.227.32:4835 , piece_id=AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.560540791 (111h46m19.112011252s) [noise=true , address=145.79.227.28:4726 , piece_id=AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.562062880 (111h46m19.113533341s) [noise=true , address=145.79.227.17:4972 , piece_id=2YAJSNY7FQXP3WCOSFQAFRWHUI3N2XGREHPZ4IBTI734FF3RXQWA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.562668593 (111h46m19.114139054s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.569385884 (111h46m19.120856355s) [noise=true , address=145.79.227.24:4581 , piece_id=CSFW35TLB7SZSYXRGPENRC4LG46JK3PV4ONSKHQEGMDDZIKYNT3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.572194983 (111h46m19.123665444s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:22.292097836 (111h46m20.843568317s) [upload=98334]: Next returned nil segment Nov 13 23:50:22.292115399 (111h46m20.84358586s) [upload=98334]: Waiting for error group managing segments... Nov 13 23:50:22.631846466 (111h46m21.183316927s) [noise=true , address=77.90.31.12:4590 , piece_id=LZ7MU5MAP6IATUSYSC7UIDST6NEJDLXDTRIL3AG4PNSPH4LLDMUA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.664080115 (111h46m21.215550566s) [noise=true , address=45.91.116.4:4939 , piece_id=BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.664601498 (111h46m21.216071959s) [noise=true , address=45.91.116.12:4225 , piece_id=B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.676547116 (111h46m21.228017577s) [noise=true , address=77.90.31.15:4699 , piece_id=KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.707487042 (111h46m21.258957493s) [noise=true , address=77.90.31.10:4517 , piece_id=UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:39.037917956 (111h46m37.589388417s) [noise=true , address=145.79.227.24:4581 , piece_id=CSFW35TLB7SZSYXRGPENRC4LG46JK3PV4ONSKHQEGMDDZIKYNT3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:41.226483506 (111h46m39.777953967s) [noise=true , address=145.79.227.23:4547 , piece_id=BLY6OGAICCKMUNKPXKDMVJ3EAAKFTSNENTHKKRP7BG2UH7CCMYMQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:41.699020421 (111h46m40.250490873s) [noise=true , address=145.79.227.31:4797 , piece_id=S4HRWXQLYLOI6YP3WMGNOCNPJAEIPQNQNBP4XGSYUZOT4NKGV7OQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:43.149006879 (111h46m41.70047733s) [noise=true , address=145.79.227.32:4835 , piece_id=AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:43.431104588 (111h46m41.98257505s) [noise=true , address=145.79.227.10:4402 , piece_id=AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:44.182010887 (111h46m42.733481348s) [noise=true , address=145.79.227.30:4762 , piece_id=WI5NJJ6PJ56HHQLO62CYNM7UGXA2ORQBK6R3GNY6O4LXD3SWLCAA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:46.418512937 (111h46m44.969983398s) [noise=true , address=145.79.227.17:4972 , piece_id=2YAJSNY7FQXP3WCOSFQAFRWHUI3N2XGREHPZ4IBTI734FF3RXQWA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:48.360715564 (111h46m46.912186025s) [noise=true , address=145.79.227.28:4726 , piece_id=AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:48.360865739 (111h46m46.91233619s) [seg_pos={0 0}, upload=98334]: Segment reached optimal threshold of 13 pieces. Nov 13 23:50:48.361094996 (111h46m46.912565457s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection Nov 13 23:50:48.361103071 (111h46m46.912573533s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361417480 (111h46m46.912887941s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection Nov 13 23:50:48.361430746 (111h46m46.912901217s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361561314 (111h46m46.913031765s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection Nov 13 23:50:48.361569741 (111h46m46.913040192s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361591251 (111h46m46.913061712s) [seg_pos={0 0}, upload=98334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 13 23:50:48.361690230 (111h46m46.913160691s) [upload=98334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 23:50:48.361696562 (111h46m46.913167023s) [upload=98334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 23:50:48.674005882 (111h46m47.225476343s) [upload=98334]: Done uploading segments... Nov 13 23:50:48.674031299 (111h46m47.225501811s) [upload=98334]: Upload finished. err: <nil> Duration:* 5.87049232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.836101767s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:51:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.836133477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...X........XX........X.............................X] duration: 30.004096757s trace: https://jaeger.dev.storj.io/trace/42cd0025356bc4b6 FAIL (2025-11-13 23:50:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, Piece ID: KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q) context deadline exceeded ======LOGS====== estItem_ObjectBegin Nov 13 23:50:20.296282661 (111h46m18.847753122s) [upload=98334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:50:20.550320931 (111h46m19.101791402s) [seg_pos={0 0}, upload=98334]: Begin upload segment... Nov 13 23:50:20.550376987 (111h46m19.101847448s) [seg_pos={0 0}, upload=98334]: Done begin upload segment. Nov 13 23:50:20.550412856 (111h46m19.101883317s) [noise=true , address=145.79.227.10:4402 , piece_id=AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.551653127 (111h46m19.103123588s) [noise=true , address=77.90.31.10:4517 , piece_id=UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.552493087 (111h46m19.103963538s) [noise=true , address=145.79.227.23:4547 , piece_id=BLY6OGAICCKMUNKPXKDMVJ3EAAKFTSNENTHKKRP7BG2UH7CCMYMQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.552926622 (111h46m19.104397094s) [noise=true , address=145.79.227.30:4762 , piece_id=WI5NJJ6PJ56HHQLO62CYNM7UGXA2ORQBK6R3GNY6O4LXD3SWLCAA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.553368324 (111h46m19.104838785s) [noise=true , address=45.91.116.12:4225 , piece_id=B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.554447308 (111h46m19.1059178s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.554916451 (111h46m19.106386902s) [noise=true , address=145.79.227.31:4797 , piece_id=S4HRWXQLYLOI6YP3WMGNOCNPJAEIPQNQNBP4XGSYUZOT4NKGV7OQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.555399110 (111h46m19.106869572s) [noise=true , address=77.90.31.15:4699 , piece_id=KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.556865883 (111h46m19.108336344s) [noise=true , address=77.90.31.12:4590 , piece_id=LZ7MU5MAP6IATUSYSC7UIDST6NEJDLXDTRIL3AG4PNSPH4LLDMUA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.558207068 (111h46m19.109677529s) [noise=true , address=45.91.116.4:4939 , piece_id=BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.559947131 (111h46m19.111417592s) [noise=true , address=145.79.227.32:4835 , piece_id=AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.560540791 (111h46m19.112011252s) [noise=true , address=145.79.227.28:4726 , piece_id=AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.562062880 (111h46m19.113533341s) [noise=true , address=145.79.227.17:4972 , piece_id=2YAJSNY7FQXP3WCOSFQAFRWHUI3N2XGREHPZ4IBTI734FF3RXQWA, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.562668593 (111h46m19.114139054s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.569385884 (111h46m19.120856355s) [noise=true , address=145.79.227.24:4581 , piece_id=CSFW35TLB7SZSYXRGPENRC4LG46JK3PV4ONSKHQEGMDDZIKYNT3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:20.572194983 (111h46m19.123665444s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Uploading piece... Nov 13 23:50:22.292097836 (111h46m20.843568317s) [upload=98334]: Next returned nil segment Nov 13 23:50:22.292115399 (111h46m20.84358586s) [upload=98334]: Waiting for error group managing segments... Nov 13 23:50:22.631846466 (111h46m21.183316927s) [noise=true , address=77.90.31.12:4590 , piece_id=LZ7MU5MAP6IATUSYSC7UIDST6NEJDLXDTRIL3AG4PNSPH4LLDMUA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.664080115 (111h46m21.215550566s) [noise=true , address=45.91.116.4:4939 , piece_id=BR4ZZXEULH5ZD4D4BTU5CK77JYP46RJ4JVWYGEHZ2F3M5OJZAINQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.664601498 (111h46m21.216071959s) [noise=true , address=45.91.116.12:4225 , piece_id=B5KBUUN7H3UO4PMPOKGCWFDN5Z5OGUBZFJADP2HYDMYSNFXWYX3Q, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.676547116 (111h46m21.228017577s) [noise=true , address=77.90.31.15:4699 , piece_id=KGZJF6Q32OUTNPEH5QQBYEWDJM3O47KGBFHYZPLTW5ZSGCT4NMXA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:22.707487042 (111h46m21.258957493s) [noise=true , address=77.90.31.10:4517 , piece_id=UP5M6Z3YJCUO7D6VD3YLXX4G7ENBALIYW6AKCBRZNYOIP7RTPKWQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:39.037917956 (111h46m37.589388417s) [noise=true , address=145.79.227.24:4581 , piece_id=CSFW35TLB7SZSYXRGPENRC4LG46JK3PV4ONSKHQEGMDDZIKYNT3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:41.226483506 (111h46m39.777953967s) [noise=true , address=145.79.227.23:4547 , piece_id=BLY6OGAICCKMUNKPXKDMVJ3EAAKFTSNENTHKKRP7BG2UH7CCMYMQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:41.699020421 (111h46m40.250490873s) [noise=true , address=145.79.227.31:4797 , piece_id=S4HRWXQLYLOI6YP3WMGNOCNPJAEIPQNQNBP4XGSYUZOT4NKGV7OQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:43.149006879 (111h46m41.70047733s) [noise=true , address=145.79.227.32:4835 , piece_id=AW67NIVOFV7W33XBBKFQSND2JIG3ZGLUNDMD6YIG56QPIVPF4JHQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:43.431104588 (111h46m41.98257505s) [noise=true , address=145.79.227.10:4402 , piece_id=AFOD5CEBMWMCUILNXUUP5NBP3NBRURKCLNNGZCO7HO6POCGHB35A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:44.182010887 (111h46m42.733481348s) [noise=true , address=145.79.227.30:4762 , piece_id=WI5NJJ6PJ56HHQLO62CYNM7UGXA2ORQBK6R3GNY6O4LXD3SWLCAA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:46.418512937 (111h46m44.969983398s) [noise=true , address=145.79.227.17:4972 , piece_id=2YAJSNY7FQXP3WCOSFQAFRWHUI3N2XGREHPZ4IBTI734FF3RXQWA, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:48.360715564 (111h46m46.912186025s) [noise=true , address=145.79.227.28:4726 , piece_id=AA6S6UEMR5BCXTW5G3L6XNNXZLU7RX77SASUGJZG7UN7XN3FZRCQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: <nil> Nov 13 23:50:48.360865739 (111h46m46.91233619s) [seg_pos={0 0}, upload=98334]: Segment reached optimal threshold of 13 pieces. Nov 13 23:50:48.361094996 (111h46m46.912565457s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48548->145.79.227.8:4329: use of closed network connection Nov 13 23:50:48.361103071 (111h46m46.912573533s) [noise=true , address=145.79.227.8:4329 , piece_id=4HH64C23Z4BVZ3SJFOO6SLMCROEK6A3W2XZ4ZO5FUZR2CC4BEM3A, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361417480 (111h46m46.912887941s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.14:4840: use of closed network connection Nov 13 23:50:48.361430746 (111h46m46.912901217s) [noise=true , address=145.79.227.14:4840 , piece_id=AWKJ2R2FGPDLENBEUFPLDUJ4QVU3FDDLTL5VBPU24CAOAX6IH5KQ, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361561314 (111h46m46.913031765s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35804->145.79.227.15:4875: use of closed network connection Nov 13 23:50:48.361569741 (111h46m46.913040192s) [noise=true , address=145.79.227.15:4875 , piece_id=I4GBOLXK7GMTPMFXWXTYS5G7G7JSBXIJYGS4JEGZ4REBIJ6ZRW3A, seg_pos={0 0}, upload=98334]: Long tail cancellation due to optimal threshold Nov 13 23:50:48.361591251 (111h46m46.913061712s) [seg_pos={0 0}, upload=98334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 13 23:50:48.361690230 (111h46m46.913160691s) [upload=98334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 23:50:48.361696562 (111h46m46.913167023s) [upload=98334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 23:50:48.674005882 (111h46m47.225476343s) [upload=98334]: Done uploading segments... Nov 13 23:50:48.674031299 (111h46m47.225501811s) [upload=98334]: Upload finished. err: <nil> Duration:* 30.004122946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.288617996s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:50:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.28865135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.919714203s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:50:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.919739091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X........XX........X.............................] duration: 5.893648374s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:49:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.893691586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.894834906s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:49:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.894850995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...X........XX........X............................] duration: 6.071022701s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:49:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.071050453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.703749519s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:48:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.70378117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X........X...X........XX........X...........................] duration: 5.97780682s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:48:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.977827029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.822092717s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:48:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.822120249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X........X...X........XX........X..........................] duration: 5.794253257s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:47:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.794280779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.792777826s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:47:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.79281112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.784884534s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:47:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.784907019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X........X...X........XX........X.........................] duration: 5.968725185s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:47:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.968760021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.707520839s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:47:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.707549885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.665324479s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:46:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.66535584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X........X...X........XX........X........................] duration: 5.929965689s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:46:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.929992429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.632805996s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:46:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.632829751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X........X...X........XX........X.......................] duration: 5.774280045s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.77430931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.663060927s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:45:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.663119208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....X........X...X........XX........X......................] duration: 6.011162458s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:45:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.011180672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.716956825s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.716986684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....X........X...X........XX........X.....................] duration: 6.119884953s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:44:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.119916081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.818646373s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:43:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.81867183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....X........X...X........XX........X....................] duration: 5.981829226s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:43:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.981855396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....X........X...X........XX........X...................] duration: 5.893858727s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:43:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.893883966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.689864728s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:43:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.6898919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X....X........X...X........XX........X..................] duration: 5.94133304s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:43:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.941367695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.758661412s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:42:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.758750932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X....X........X...X........XX........X.................] duration: 5.950495671s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:41:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.950522601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X....X........X...X........XX........X................] duration: 6.010898554s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:41:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.010921497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X....X........X...X........XX........X...............] duration: 5.829882747s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.829911562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.833934793s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:40:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.833961383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X....X........X...X........XX........X..............] duration: 5.944733299s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:40:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.94476047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.837364856s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:40:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.837451901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X....X........X...X........XX........X.............] duration: 5.940572847s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:40:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.940604726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....X........X...X........XX........X............] duration: 6.072612941s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:39:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.072639661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.838187137s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:39:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.838228055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X....X........X...X........XX........X...........] duration: 5.887867835s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:38:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.887897392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.825728727s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:38:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.825773151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X....X........X...X........XX........X..........] duration: 5.863102253s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:38:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.863132379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.824287585s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:37:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.824314667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X............X....X........X...X........XX........X.........] duration: 5.913406314s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:37:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.913434258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.638958344s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:36:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.638989172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.809120591s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:36:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.809148594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX............X....X........X...X........XX........X........] duration: 5.992651615s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:36:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.992673847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.837629515s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:36:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.837657549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX............X....X........X...X........XX........X.......] duration: 5.920878687s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:36:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.920902913s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)