Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 5.657295581s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:47:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.657327062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 5.662175909s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.662206418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 5.631752901s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:45:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.631780932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration: 5.732957085s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:45:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.733010006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration: 5.53016424s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:44:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.530193967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration: 5.71854427s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:43:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.718572875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration: 5.637457915s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:43:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.637661072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration: 5.698452309s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:42:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.698495131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration: 5.669902124s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:41:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.669927132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration: 5.572724024s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:40:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.572768497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration: 5.644058433s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:39:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.64415579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration: 5.566305051s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:38:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.566331251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration: 5.760631142s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:38:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.760661088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration: 5.669642188s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:37:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.669669911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration: 5.678325435s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.678379368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration: 5.788311055s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:36:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.788339288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration: 5.704563934s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.704591328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration: 5.89171068s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:34:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.891765104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................XXXXX.......................................] duration: 5.683382805s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:33:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.683410077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................XXXXX......................................] duration: 5.808309494s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:32:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.808343308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................XXXXX.....................................] duration: 5.726554326s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:31:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.726584603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................XXXXX....................................] duration: 5.740929047s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:30:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.740968873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................XXXXX...................................] duration: 5.678925516s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:29:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.678960092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration: 5.579583503s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:28:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.57961318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration: 5.734776346s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:27:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.734806724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration: 5.633762599s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:26:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.633796291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration: 6.016761448s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:26:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.016792997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration: 5.748795633s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:25:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.74882537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration: 5.685007059s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:24:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.685040042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration: 5.631881838s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:23:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.631993752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration: 5.627410733s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:22:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.627436201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration: 5.688498104s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:21:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.688528684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration: 5.522062478s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:20:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.522087927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration: 5.681990893s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:20:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.682023947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................XXXXX.......................] duration: 5.554276299s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:19:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.554306606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................XXXXX......................] duration: 5.589046047s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:18:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.589079399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................XXXXX.....................] duration: 5.504005963s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.504025681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................XXXXX....................] duration: 5.866389584s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:17:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.866421275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration: 5.758106685s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:16:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.758150098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration: 5.666982305s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:15:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.667034596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration: 5.633063857s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:14:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.633106157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration: 5.582563461s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:13:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.582588048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration: 5.451469314s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:13:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.451580415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration: 5.668923333s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.66895343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration: 5.653507878s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:11:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.653587211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration: 5.635440814s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:11:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.635471924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration: 5.669932356s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:10:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.669964308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration: 5.851124508s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:10:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.85116259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXXXX.........] duration: 5.642639748s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:09:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.642669784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXXXX........] duration: 5.678026263s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:08:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.678058102s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)