Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XXXXX.......] duration: 5.740570492s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:07:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.740598605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XXXXX......] duration: 5.657886179s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:06:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.657944168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration: 5.616735611s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:06:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.616764768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 5.702353283s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:05:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.702382679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 5.745618913s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:04:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.745662446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 5.765105249s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.765131118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 38.212894335s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:03:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 38.212922818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.245341191s trace: https://jaeger.dev.storj.io/trace/56ffc589e0c5772a FAIL (2025-11-30 10:02:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Nov 30 10:01:24.460646521 (505h57m23.012116982s) [seg_pos={0 0}, upload=445542]: Done begin upload segment. Nov 30 10:01:24.460733988 (505h57m23.012204439s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.460751932 (505h57m23.012222393s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461422129 (505h57m23.01289258s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.461580209 (505h57m23.01305066s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462103736 (505h57m23.013574197s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462448353 (505h57m23.013918804s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.462775867 (505h57m23.014246328s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463092690 (505h57m23.014563151s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463418610 (505h57m23.014889061s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.463804515 (505h57m23.015274976s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464148560 (505h57m23.015619021s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464557329 (505h57m23.01602779s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.464848092 (505h57m23.016318553s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465132794 (505h57m23.016603255s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465426274 (505h57m23.016896735s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.465704784 (505h57m23.017175235s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470301647 (505h57m23.021772098s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:24.470730052 (505h57m23.022200504s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Uploading piece... Nov 30 10:01:25.756204017 (505h57m24.307674478s) [upload=445542]: Next returned nil segment Nov 30 10:01:25.756220048 (505h57m24.307690509s) [upload=445542]: Waiting for error group managing segments... Nov 30 10:01:26.077787818 (505h57m24.629258279s) [noise=true , address=64.49.13.26:4950 , piece_id=6TNACF4ITUDIX2A6ISYVS2P2QIQ2NYMXEGYZGVLMOT2SCPQHSF6Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.163135087 (505h57m24.714605538s) [noise=true , address=64.49.13.137:4495 , piece_id=V4TGWH5YODLMGTVWM66T7PAEGQVNDEDIBV3TJL5BZ2OGQA4K4MDA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.189162669 (505h57m24.74063313s) [noise=true , address=193.149.164.143:4698 , piece_id=YPPZUYMPMTRDBJMKLBJHNCKN7325AFJYBYLWJTJ2EA4YKVFDAVXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.357484278 (505h57m24.908954739s) [noise=true , address=145.79.227.14:4848 , piece_id=Y2WS7DW5M4WZWQR6Y5D4KSMY32AQDMDUSSFISQNIDHBGIRUNE42Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.426798566 (505h57m24.978269027s) [noise=true , address=45.91.116.10:4087 , piece_id=AU5VESS732DYDLJTIDIFNKKV622LZVQ3E66IK4IOM5YXFQ535SIQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.450260893 (505h57m25.001731354s) [noise=true , address=45.91.116.11:4140 , piece_id=NKU4O7JCIIFBL73P7TOVG2EJ5GQE7TIQ6OQJP4FM7GXN5XDCZI2A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.477387297 (505h57m25.028857748s) [noise=true , address=45.91.116.9:4268 , piece_id=IXJDEQVX3CRH4THORR5SXYUZNCJYUBWPXKMGFTMTGHBMKLP2UCYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.486235215 (505h57m25.037705676s) [noise=true , address=45.91.116.13:4182 , piece_id=T34B7I4XETMSLXRL6DGP35GH7MW4GH2H6SIBLB4OTSM6VD4YRP5A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.507726178 (505h57m25.059196639s) [noise=true , address=45.91.116.7:4036 , piece_id=MVTJGTHGP4SGOW3JO7J5OVHAV7CQMMOL6OYJT6JABV7WRQX5WGXA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.515452632 (505h57m25.066923113s) [noise=true , address=45.91.116.14:4205 , piece_id=GDHENKRVLB4GPDSX7FI3HUYI5MXIIJ27HVMPMLO422CI5UQCCW7A, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:01:26.516645543 (505h57m25.068116014s) [noise=true , address=45.91.116.4:4948 , piece_id=CTCJSYVXNTHY25QDOWOUMTQWUZL6QLKBUKSNAH34VGVGOUCPUBPQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: <nil> Nov 30 10:02:23.512289546 (505h58m22.063759997s) [noise=true , address=77.90.31.11:4564 , piece_id=VGTWIP5OAQWDIV5XON2VF3K5F2M2D3MEWTLNRGZRQYEDTYPUSPRQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512338739 (505h58m22.0638092s) [noise=true , address=77.90.31.14:4690 , piece_id=MUCEBQP3UUWPFQBIRRJD7NVMGUG2ZJUAK2ZK5FOWABVZWRBEIVYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512474608 (505h58m22.063945069s) [noise=true , address=77.90.31.12:4600 , piece_id=ZZTKLGOQRZOGPJ5GTHCVUUJW47RS4CVC7RM2ZJRNX6ZPNFEC4EYQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512632289 (505h58m22.06410275s) [noise=true , address=77.90.31.16:4753 , piece_id=PAEPZ4JV7BA5OQ4KNGXL3S7R3B72CVYSOWCYRDMWAXGOEXTMZO7Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, address:77.90.31.16:4753): protocol: rpc: dial tcp 77.90.31.16:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512871675 (505h58m22.064342126s) [noise=true , address=77.90.31.17:4773 , piece_id=XS3JA3ZEMXFYLQRH3YAHBYRZGW72JHWHGU2OHJVHPRZ5S2LGBXJQ, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.512878688 (505h58m22.064349149s) [noise=true , address=77.90.31.10:4534 , piece_id=TKCPHE3RDNQ62M45UOIRENMUQA5UZOVA32A6K3KHJ5FAX5ZUTMQA, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: rpc: dial tcp 77.90.31.10:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513010088 (505h58m22.064480539s) [noise=true , address=77.90.31.21:4902 , piece_id=QW7C6PLHHINV475F4Q2Z2MIEGNHTY6WSPL4O7DWMT2AX3AR3NJ5Q, seg_pos={0 0}, upload=445542]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:48248->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:02:23.513044093 (505h58m22.064514544s) [seg_pos={0 0}, upload=445542]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 10:02:23.755707764 (505h58m22.307178226s) [upload=445542]: Done uploading segments... Nov 30 10:02:23.755752459 (505h58m22.30722292s) [upload=445542]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.245355378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.254419764s trace: https://jaeger.dev.storj.io/trace/1d4566856373f184 FAIL (2025-11-30 10:01:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== s={0 0}, upload=445527]: Begin upload segment... Nov 30 10:00:23.977776652 (505h56m22.529247113s) [seg_pos={0 0}, upload=445527]: Done begin upload segment. Nov 30 10:00:23.977996089 (505h56m22.52946655s) [noise=true , address=77.90.31.11:4557 , piece_id=4WHLIUJZMZ2C5K75NN6MQQWRHDETNIDADJT5KADEYFOSPTDOOOVA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.979000021 (505h56m22.530470472s) [noise=true , address=77.90.31.17:4802 , piece_id=UCAJBJZUNXPTZOGMTQNCBSDLVDHISEZIOHTOOLFA7VGAVU6ZPN2Q, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.979418989 (505h56m22.53088945s) [noise=true , address=45.91.116.12:4233 , piece_id=LLIFPLG6E5Z6OMY6MHA3WDAEFRPJR2YR6IM7GMQWT47QPHEQYCFA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.979826344 (505h56m22.531296805s) [noise=true , address=45.91.116.5:4304 , piece_id=M54GEODGRU4ESGAMABXQVGZGMRI5PQNMHARBN36CFLHCOUQBR2LA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.980226757 (505h56m22.531697218s) [noise=true , address=77.90.31.19:4866 , piece_id=JOSSVFBVBFG4UTBTHIFZLC2S5O6AITGIQ3RJU572FWTGTEUI4QOA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.980657536 (505h56m22.532128008s) [noise=true , address=145.79.227.3:4166 , piece_id=YBVX5NBOOK4CLPZJXN434XFUWCRL4V4UQRQB44BFHFYXLVC3QWCA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.981278419 (505h56m22.53274888s) [noise=true , address=45.91.116.7:4012 , piece_id=UT236NDRMNA33TCI4TIP5IGHNWQ2J2GWB2KV6WRKBUH7V6VVRBQA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.981712706 (505h56m22.533183167s) [noise=true , address=77.90.31.10:4529 , piece_id=HZY3AZLXCSF4GH6263P5B5JFQZUGXXMRRQ4ZR5H6PFK4MCQLTRWA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.982220423 (505h56m22.533690884s) [noise=true , address=45.91.116.9:4285 , piece_id=DSWGGZ3LYD5FJCJLDC7TRPO4PTCPCGXNGEP4IQFF6E6D4SGPD6YA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.982629842 (505h56m22.534100304s) [noise=true , address=77.90.31.16:4763 , piece_id=47B5OW326KMJPRSGYVTRPVYMC23ZRRQB4TIXCSD5WSRFESLV5Q4A, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.983051405 (505h56m22.534521866s) [noise=true , address=77.90.31.21:4895 , piece_id=DXVBZQ4Y3OUQPEAX67DAEO3FF54DQJV32K4WI2D6KNORGEHN5D7Q, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.983487726 (505h56m22.534958187s) [noise=true , address=45.91.116.13:4167 , piece_id=BTREKDNMA5BH5PPY3PMKCPFKKMIPUCWYJGBYN7I5D3Y2RCJ77BJQ, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.983919538 (505h56m22.535389999s) [noise=true , address=77.90.31.13:4658 , piece_id=CR7QSLFQDG73FZIV7K3QGWOYFY7QTN74E654FETZOPOPY6X4ESQA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.984330010 (505h56m22.535800471s) [noise=true , address=77.90.31.14:4692 , piece_id=XUGXILRLASWEQM5LDX52BAVBTKNBVHF2DNMAJMCEXUY3HVWTNQLA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.984765569 (505h56m22.53623602s) [noise=true , address=45.91.116.11:4140 , piece_id=3PO75FED5H2HQ5MZVUCSOWMLNWX7Y7SNB2H6XGSRKOP22WFAW7KA, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.985234272 (505h56m22.536704743s) [noise=true , address=193.149.164.143:4696 , piece_id=NOKXXKJZNKR2JZAGD6XYCTTKJV22CZRDZBLCA4NDMPQYOZ7K7XDQ, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.985697163 (505h56m22.537167624s) [noise=true , address=77.90.31.15:4717 , piece_id=5NRK4NLOAM66T3PX75NRDHVQQ2O47RVTQIS2NF5VQ4H3RMRHAQWQ, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:23.991777181 (505h56m22.543247632s) [noise=true , address=145.79.227.14:4855 , piece_id=RCJUNGAXOIAXGYPQA743ZWNQZLT7GYSLQH3QLJP3NT3W6V7L4G6A, seg_pos={0 0}, upload=445527]: Uploading piece... Nov 30 10:00:25.320635659 (505h56m23.87210612s) [upload=445527]: Next returned nil segment Nov 30 10:00:25.320659705 (505h56m23.872130166s) [upload=445527]: Waiting for error group managing segments... Nov 30 10:00:25.651932256 (505h56m24.203402717s) [noise=true , address=193.149.164.143:4696 , piece_id=NOKXXKJZNKR2JZAGD6XYCTTKJV22CZRDZBLCA4NDMPQYOZ7K7XDQ, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:25.887355760 (505h56m24.438826231s) [noise=true , address=145.79.227.14:4855 , piece_id=RCJUNGAXOIAXGYPQA743ZWNQZLT7GYSLQH3QLJP3NT3W6V7L4G6A, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:25.892049298 (505h56m24.443519759s) [noise=true , address=145.79.227.3:4166 , piece_id=YBVX5NBOOK4CLPZJXN434XFUWCRL4V4UQRQB44BFHFYXLVC3QWCA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:25.957835899 (505h56m24.50930635s) [noise=true , address=45.91.116.11:4140 , piece_id=3PO75FED5H2HQ5MZVUCSOWMLNWX7Y7SNB2H6XGSRKOP22WFAW7KA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:25.960702307 (505h56m24.512172768s) [noise=true , address=45.91.116.5:4304 , piece_id=M54GEODGRU4ESGAMABXQVGZGMRI5PQNMHARBN36CFLHCOUQBR2LA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:25.984438505 (505h56m24.535908976s) [noise=true , address=45.91.116.7:4012 , piece_id=UT236NDRMNA33TCI4TIP5IGHNWQ2J2GWB2KV6WRKBUH7V6VVRBQA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:25.990942952 (505h56m24.542413413s) [noise=true , address=45.91.116.9:4285 , piece_id=DSWGGZ3LYD5FJCJLDC7TRPO4PTCPCGXNGEP4IQFF6E6D4SGPD6YA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:25.994728229 (505h56m24.54619869s) [noise=true , address=45.91.116.13:4167 , piece_id=BTREKDNMA5BH5PPY3PMKCPFKKMIPUCWYJGBYN7I5D3Y2RCJ77BJQ, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:00:26.037246152 (505h56m24.588716613s) [noise=true , address=45.91.116.12:4233 , piece_id=LLIFPLG6E5Z6OMY6MHA3WDAEFRPJR2YR6IM7GMQWT47QPHEQYCFA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: <nil> Nov 30 10:01:23.031833008 (505h57m21.583303469s) [noise=true , address=77.90.31.11:4557 , piece_id=4WHLIUJZMZ2C5K75NN6MQQWRHDETNIDADJT5KADEYFOSPTDOOOVA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: rpc: dial tcp 77.90.31.11:4557: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:01:23.032312211 (505h57m21.583782672s) [noise=true , address=77.90.31.17:4802 , piece_id=UCAJBJZUNXPTZOGMTQNCBSDLVDHISEZIOHTOOLFA7VGAVU6ZPN2Q, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:12hiNtWkLwWMYr8rPN9ukEKQou5Eko2fD9ScXsqBKk5oX5wxgHy, address:77.90.31.17:4802): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.032473518 (505h57m21.583943979s) [noise=true , address=77.90.31.19:4866 , piece_id=JOSSVFBVBFG4UTBTHIFZLC2S5O6AITGIQ3RJU572FWTGTEUI4QOA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:12uCbmPHVXpukjQahs2G6oh7VY2zDMooMAbdgSyHN7Biuesmz3A, address:77.90.31.19:4866): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.032620157 (505h57m21.584090608s) [noise=true , address=77.90.31.14:4692 , piece_id=XUGXILRLASWEQM5LDX52BAVBTKNBVHF2DNMAJMCEXUY3HVWTNQLA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T, address:77.90.31.14:4692): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.032772627 (505h57m21.584243088s) [noise=true , address=77.90.31.21:4895 , piece_id=DXVBZQ4Y3OUQPEAX67DAEO3FF54DQJV32K4WI2D6KNORGEHN5D7Q, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.033347302 (505h57m21.584817763s) [noise=true , address=77.90.31.15:4717 , piece_id=5NRK4NLOAM66T3PX75NRDHVQQ2O47RVTQIS2NF5VQ4H3RMRHAQWQ, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.033506045 (505h57m21.584976506s) [noise=true , address=77.90.31.16:4763 , piece_id=47B5OW326KMJPRSGYVTRPVYMC23ZRRQB4TIXCSD5WSRFESLV5Q4A, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:12Bn4bKB4U5JidFqnXuPTncmNPGi72tgbL5YmYBRbbJ6ckqo35d, address:77.90.31.16:4763): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.034050621 (505h57m21.585521082s) [noise=true , address=77.90.31.10:4529 , piece_id=HZY3AZLXCSF4GH6263P5B5JFQZUGXXMRRQ4ZR5H6PFK4MCQLTRWA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.034186810 (505h57m21.585657272s) [noise=true , address=77.90.31.13:4658 , piece_id=CR7QSLFQDG73FZIV7K3QGWOYFY7QTN74E654FETZOPOPY6X4ESQA, seg_pos={0 0}, upload=445527]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: context deadline exceeded; context deadline exceeded Nov 30 10:01:23.034219693 (505h57m21.585690154s) [seg_pos={0 0}, upload=445527]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Nov 30 10:01:23.283591949 (505h57m21.83506242s) [upload=445527]: Done uploading segments... Nov 30 10:01:23.283633679 (505h57m21.83510413s) [upload=445527]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.254445494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.255048505s trace: https://jaeger.dev.storj.io/trace/27fabf18c682f652 FAIL (2025-11-30 10:00:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== =445507]: Uploading piece... Nov 30 09:59:12.886909813 (505h55m11.438380274s) [noise=true , address=45.91.116.10:4092 , piece_id=NQ5GZP3ZLXLHFG7FWPN573A7NPOQCEQREBRCI3UBXVXK454B64PQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.887294345 (505h55m11.438764807s) [noise=true , address=77.90.31.13:4642 , piece_id=XJGBY4DWM5EHFR6DGQBRONNLXEZYQXHNCXAMBBYSMNBK2IO5ABWA, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.887529774 (505h55m11.439000235s) [noise=true , address=77.90.31.17:4796 , piece_id=Z6S5M6I5Z52757O3EEZBUPFUXCNS3OQ65SVTESNDK2NEK3GBMWIA, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.887911561 (505h55m11.439382022s) [noise=true , address=45.91.116.14:4218 , piece_id=7YJYBSKOXIB3A643FXRZQ4V4MRXPMCJB6RL5TQJVDRN7MYNB5UIQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.888204108 (505h55m11.439674569s) [noise=true , address=45.91.116.9:4282 , piece_id=PHDDDYPEPTXKPTPA7MYSDRXU6L2KAMPSIUPKRFJYLEGSHEGD7P2A, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.888486195 (505h55m11.439956646s) [noise=true , address=45.91.116.6:4982 , piece_id=VM6MEGICFMLT5KXVWUVSW4KZDSQGH252F6ZTK345G77LZPNQEW2A, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.888827936 (505h55m11.440298397s) [noise=true , address=193.149.164.143:4698 , piece_id=R7L45IWGRPHWUYDSFV4SDUAVCJU5VBY5UM3IFM3QFU67TCGVAVZQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.889134690 (505h55m11.440605151s) [noise=true , address=64.49.13.155:4916 , piece_id=3F4BMA3SUCHRAH35POACCHYY3YI4JZ5SV2GUHHYJ6VTXQE3IM3AA, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.889550772 (505h55m11.441021223s) [noise=true , address=77.90.31.16:4757 , piece_id=S46NMOWNNCEELOMBEOWTA4BL3VXRPZDXGBRKSNF66L7Z7RHJNI6Q, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.889941125 (505h55m11.441411586s) [noise=true , address=45.91.116.8:4058 , piece_id=VNCZDCHXNW6RWP2U5E2L3MAOVLYRXZJZIZOOMIVCN5JXQDD4JJGQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.890421911 (505h55m11.441892362s) [noise=true , address=77.90.31.11:4572 , piece_id=YAY2L6PUX4HPWPLRSBYZ3T5FWH6FUEHNRLKSDCGHDEWBQLSTIBPQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.890941380 (505h55m11.442411841s) [noise=true , address=77.90.31.18:4811 , piece_id=LD2Z5OPNIC53LHCBET6TS443A7ZPA3OAZV74G7BIFG6SYAGM5AYA, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.891445982 (505h55m11.442916433s) [noise=true , address=77.90.31.15:4713 , piece_id=LBMBAPYQ2EURV7LWX3FVVBVMSR5762QUCNKJBJ4SQYYASOETZJVQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.892911302 (505h55m11.444381763s) [noise=true , address=77.90.31.10:4533 , piece_id=4JW7QARW5KB54J6UYKHYD2IC2PVRCSXSIYC5O3ECNXC6BE3J4HMQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.894884158 (505h55m11.446354609s) [noise=true , address=64.49.13.169:4705 , piece_id=XNZKJG22RN3LSZZUAEBPWL4DBDKHQOWVSOBPUSCNSAYVWH3I72MQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.896352253 (505h55m11.447822744s) [noise=true , address=64.49.13.170:4718 , piece_id=WWOY3G56SXNA4SYV3XKJ23WJZZLYIPDF364BF3YQUVCTSLRGVDYA, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:12.897708095 (505h55m11.449178556s) [noise=true , address=77.90.31.19:4860 , piece_id=SI7BRI7V3FAHBZTDI4ZYSK5D2FTK22FJ2JYBYFOSA5ECZELYMKLQ, seg_pos={0 0}, upload=445507]: Uploading piece... Nov 30 09:59:14.251182988 (505h55m12.802653449s) [upload=445507]: Next returned nil segment Nov 30 09:59:14.251200060 (505h55m12.802670521s) [upload=445507]: Waiting for error group managing segments... Nov 30 09:59:14.595403093 (505h55m13.146873544s) [noise=true , address=64.49.13.170:4718 , piece_id=WWOY3G56SXNA4SYV3XKJ23WJZZLYIPDF364BF3YQUVCTSLRGVDYA, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.670676870 (505h55m13.222147321s) [noise=true , address=64.49.13.169:4705 , piece_id=XNZKJG22RN3LSZZUAEBPWL4DBDKHQOWVSOBPUSCNSAYVWH3I72MQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.695933834 (505h55m13.247404285s) [noise=true , address=193.149.164.143:4698 , piece_id=R7L45IWGRPHWUYDSFV4SDUAVCJU5VBY5UM3IFM3QFU67TCGVAVZQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.696423336 (505h55m13.247893797s) [noise=true , address=64.49.13.155:4916 , piece_id=3F4BMA3SUCHRAH35POACCHYY3YI4JZ5SV2GUHHYJ6VTXQE3IM3AA, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.825407539 (505h55m13.376878s) [noise=true , address=45.91.116.8:4058 , piece_id=VNCZDCHXNW6RWP2U5E2L3MAOVLYRXZJZIZOOMIVCN5JXQDD4JJGQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.884965069 (505h55m13.43643552s) [noise=true , address=45.91.116.6:4982 , piece_id=VM6MEGICFMLT5KXVWUVSW4KZDSQGH252F6ZTK345G77LZPNQEW2A, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.900077024 (505h55m13.451547485s) [noise=true , address=45.91.116.9:4282 , piece_id=PHDDDYPEPTXKPTPA7MYSDRXU6L2KAMPSIUPKRFJYLEGSHEGD7P2A, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.902226638 (505h55m13.453697099s) [noise=true , address=45.91.116.10:4092 , piece_id=NQ5GZP3ZLXLHFG7FWPN573A7NPOQCEQREBRCI3UBXVXK454B64PQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.908802279 (505h55m13.46027274s) [noise=true , address=45.91.116.14:4218 , piece_id=7YJYBSKOXIB3A643FXRZQ4V4MRXPMCJB6RL5TQJVDRN7MYNB5UIQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 09:59:14.919413916 (505h55m13.470884377s) [noise=true , address=45.91.116.7:4027 , piece_id=GTDBALT7DH4QINHEYU4RSELDO3JOO7DCIC363IJCF34UB2P7PGHQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: <nil> Nov 30 10:00:11.923772997 (505h56m10.475243458s) [noise=true , address=77.90.31.18:4811 , piece_id=LD2Z5OPNIC53LHCBET6TS443A7ZPA3OAZV74G7BIFG6SYAGM5AYA, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6, address:77.90.31.18:4811): protocol: rpc: dial tcp 77.90.31.18:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.924365645 (505h56m10.475836106s) [noise=true , address=77.90.31.11:4572 , piece_id=YAY2L6PUX4HPWPLRSBYZ3T5FWH6FUEHNRLKSDCGHDEWBQLSTIBPQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.924860107 (505h56m10.476330578s) [noise=true , address=77.90.31.15:4713 , piece_id=LBMBAPYQ2EURV7LWX3FVVBVMSR5762QUCNKJBJ4SQYYASOETZJVQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: write tcp 188.166.225.170:54496->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.924935200 (505h56m10.476405661s) [noise=true , address=77.90.31.10:4533 , piece_id=4JW7QARW5KB54J6UYKHYD2IC2PVRCSXSIYC5O3ECNXC6BE3J4HMQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: rpc: dial tcp 77.90.31.10:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.925553647 (505h56m10.477024109s) [noise=true , address=77.90.31.13:4642 , piece_id=XJGBY4DWM5EHFR6DGQBRONNLXEZYQXHNCXAMBBYSMNBK2IO5ABWA, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 188.166.225.170:43516->77.90.31.13:4642: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.926136818 (505h56m10.477607269s) [noise=true , address=77.90.31.19:4860 , piece_id=SI7BRI7V3FAHBZTDI4ZYSK5D2FTK22FJ2JYBYFOSA5ECZELYMKLQ, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, address:77.90.31.19:4860): protocol: rpc: dial tcp 77.90.31.19:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.926457428 (505h56m10.47792789s) [noise=true , address=77.90.31.17:4796 , piece_id=Z6S5M6I5Z52757O3EEZBUPFUXCNS3OQ65SVTESNDK2NEK3GBMWIA, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: rpc: dial tcp 77.90.31.17:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.926672287 (505h56m10.478142748s) [noise=true , address=77.90.31.16:4757 , piece_id=S46NMOWNNCEELOMBEOWTA4BL3VXRPZDXGBRKSNF66L7Z7RHJNI6Q, seg_pos={0 0}, upload=445507]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 188.166.225.170:46666->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 10:00:11.926712143 (505h56m10.478182604s) [seg_pos={0 0}, upload=445507]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 30 10:00:12.175557184 (505h56m10.727027655s) [upload=445507]: Done uploading segments... Nov 30 10:00:12.175606959 (505h56m10.72707742s) [upload=445507]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.255061429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.25941952s trace: https://jaeger.dev.storj.io/trace/70472640f5960378 FAIL (2025-11-30 09:59:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 95 (505h54m10.942673846s) [seg_pos={0 0}, upload=445495]: Done begin upload segment. Nov 30 09:58:12.391235226 (505h54m10.942705687s) [noise=true , address=77.90.31.14:4688 , piece_id=QDN5BJCHJQW5VU52S5P3Q7S6QNTWECVYJCPQEAFLTYZSUDOZBB7Q, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.393958982 (505h54m10.945429453s) [noise=true , address=77.90.31.13:4654 , piece_id=T2SETIF7KDES4DNH6HKWE6OT745ETU2ERSL5SDEROJJDME64UYYQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.394734829 (505h54m10.94620529s) [noise=true , address=145.79.227.18:4019 , piece_id=PZTSBPN6ZYAC3MAODKYVUK5UTQ75SNXJTB634LWVRXS7FYULBD3A, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.395223500 (505h54m10.946693961s) [noise=true , address=77.90.31.21:4892 , piece_id=JJUBUY22HECNQA5U7VJWBISZZJPWI73JYE2OOYSXACWSGYWEYIFA, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.395694447 (505h54m10.947164908s) [noise=true , address=77.90.31.15:4716 , piece_id=TL3RLUBVT43M6QHHFMDAQDYLVLK5DOPDCCR2J4OGG63D5XX3T6VQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.396124335 (505h54m10.947594796s) [noise=true , address=64.49.13.169:4709 , piece_id=WF3SWUFVXLHAHF6IHPAVKGMZFDAK6PLZQOQSSM3J65F65JGP4PBQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.396550667 (505h54m10.948021128s) [noise=true , address=45.91.116.9:4273 , piece_id=WZFNWB36ZT5TBHM3JVORQRHAC6IA6AXZSOZNNO4SLYT2CWPW7KLQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.397007767 (505h54m10.948478228s) [noise=true , address=77.90.31.12:4623 , piece_id=KV3HWTH37MNBNUL3QZKBIBWFSKZVTA4OU4CSU2JBFL633DUMR7TQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.397470248 (505h54m10.94894071s) [noise=true , address=64.49.13.169:4705 , piece_id=FCAWKSRNSE3DCY2VEDHSUJCYCYEFGFRZZ3RQLNHW44FXBN4YDJOQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.398262397 (505h54m10.949732858s) [noise=true , address=77.90.31.16:4739 , piece_id=ZCP2QMOZVLVWASVFOI5T4XD5I4I6PTA5GTDTNELUWPOUKNGIV7XA, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.398284088 (505h54m10.949754549s) [noise=true , address=45.91.116.6:4999 , piece_id=SWKMVKBYF2UWJ7RNPFWE5KQNO4Q3NDLYXJGFJP4HEVSGZQINWN7A, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.398847050 (505h54m10.950317511s) [noise=true , address=45.91.116.13:4185 , piece_id=RMMSRBGWQGFTPNJYRXBTE4K3PMU5AITVFRKFBHWQ2Y4V47WGJADA, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.399388471 (505h54m10.950858932s) [noise=true , address=45.91.116.14:4218 , piece_id=QYS6UEYO4DADX7GJ72PTDEZH6T2MUNZDW37SKVQIHILZO4GWJIEQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.399916576 (505h54m10.951387037s) [noise=true , address=77.90.31.10:4548 , piece_id=WXFETS7ZJSMLETSLWRARVCLDT3ROABGAGELADLTMTZGF76IPO3VA, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.400465893 (505h54m10.951936344s) [noise=true , address=145.79.227.15:4899 , piece_id=YHV2BR7E7OEO2H7QJNJ2JMKFCOX632VN3RYMYS2SBWNHK2JYWJ2A, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.401002475 (505h54m10.952472936s) [noise=true , address=145.79.227.14:4842 , piece_id=FEWSTQVV6VPV6VSESE5CGPWHZZFBYACVSP5BWDBRBM5YL2XG4HIA, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.401504270 (505h54m10.952974731s) [noise=true , address=45.91.116.8:4069 , piece_id=YCBVIVQTOTVCUN2RDOG3KPAAYKD4PFVZG6CMCGXJLGXW3LRTY7ZQ, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:12.403243231 (505h54m10.954713692s) [noise=true , address=64.49.13.169:4715 , piece_id=5DS3PVGWLZ3EUCX7JGT243YEISAYLPWOMWP55X7NWLFD5CWGUD6A, seg_pos={0 0}, upload=445495]: Uploading piece... Nov 30 09:58:13.703897273 (505h54m12.255367734s) [upload=445495]: Next returned nil segment Nov 30 09:58:13.703937589 (505h54m12.25540805s) [upload=445495]: Waiting for error group managing segments... Nov 30 09:58:14.279847269 (505h54m12.83131773s) [noise=true , address=145.79.227.18:4019 , piece_id=PZTSBPN6ZYAC3MAODKYVUK5UTQ75SNXJTB634LWVRXS7FYULBD3A, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:14.279974722 (505h54m12.831445183s) [noise=true , address=45.91.116.9:4273 , piece_id=WZFNWB36ZT5TBHM3JVORQRHAC6IA6AXZSOZNNO4SLYT2CWPW7KLQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:14.292652873 (505h54m12.844123344s) [noise=true , address=145.79.227.15:4899 , piece_id=YHV2BR7E7OEO2H7QJNJ2JMKFCOX632VN3RYMYS2SBWNHK2JYWJ2A, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:14.304941143 (505h54m12.856411604s) [noise=true , address=145.79.227.14:4842 , piece_id=FEWSTQVV6VPV6VSESE5CGPWHZZFBYACVSP5BWDBRBM5YL2XG4HIA, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:14.340565418 (505h54m12.892035879s) [noise=true , address=45.91.116.13:4185 , piece_id=RMMSRBGWQGFTPNJYRXBTE4K3PMU5AITVFRKFBHWQ2Y4V47WGJADA, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:14.371082730 (505h54m12.922553191s) [noise=true , address=45.91.116.14:4218 , piece_id=QYS6UEYO4DADX7GJ72PTDEZH6T2MUNZDW37SKVQIHILZO4GWJIEQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:14.415627352 (505h54m12.967097813s) [noise=true , address=45.91.116.6:4999 , piece_id=SWKMVKBYF2UWJ7RNPFWE5KQNO4Q3NDLYXJGFJP4HEVSGZQINWN7A, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:14.428341451 (505h54m12.979811902s) [noise=true , address=45.91.116.8:4069 , piece_id=YCBVIVQTOTVCUN2RDOG3KPAAYKD4PFVZG6CMCGXJLGXW3LRTY7ZQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:15.284212816 (505h54m13.835683277s) [noise=true , address=64.49.13.169:4709 , piece_id=WF3SWUFVXLHAHF6IHPAVKGMZFDAK6PLZQOQSSM3J65F65JGP4PBQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:15.378870775 (505h54m13.930341236s) [noise=true , address=64.49.13.169:4705 , piece_id=FCAWKSRNSE3DCY2VEDHSUJCYCYEFGFRZZ3RQLNHW44FXBN4YDJOQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:58:15.379990136 (505h54m13.931460587s) [noise=true , address=64.49.13.169:4715 , piece_id=5DS3PVGWLZ3EUCX7JGT243YEISAYLPWOMWP55X7NWLFD5CWGUD6A, seg_pos={0 0}, upload=445495]: Done uploading piece. err: <nil> Nov 30 09:59:11.436833041 (505h55m9.988303512s) [noise=true , address=77.90.31.10:4548 , piece_id=WXFETS7ZJSMLETSLWRARVCLDT3ROABGAGELADLTMTZGF76IPO3VA, seg_pos={0 0}, upload=445495]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: rpc: dial tcp 77.90.31.10:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:11.436854131 (505h55m9.988324603s) [noise=true , address=77.90.31.13:4654 , piece_id=T2SETIF7KDES4DNH6HKWE6OT745ETU2ERSL5SDEROJJDME64UYYQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: rpc: dial tcp 77.90.31.13:4654: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:11.437045105 (505h55m9.988515566s) [noise=true , address=77.90.31.14:4688 , piece_id=QDN5BJCHJQW5VU52S5P3Q7S6QNTWECVYJCPQEAFLTYZSUDOZBB7Q, seg_pos={0 0}, upload=445495]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:11.437097505 (505h55m9.988567966s) [noise=true , address=77.90.31.21:4892 , piece_id=JJUBUY22HECNQA5U7VJWBISZZJPWI73JYE2OOYSXACWSGYWEYIFA, seg_pos={0 0}, upload=445495]: Done uploading piece. err: ecclient: upload failed (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa, address:77.90.31.21:4892): protocol: rpc: dial tcp 77.90.31.21:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:11.437211161 (505h55m9.988681632s) [noise=true , address=77.90.31.15:4716 , piece_id=TL3RLUBVT43M6QHHFMDAQDYLVLK5DOPDCCR2J4OGG63D5XX3T6VQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: ecclient: upload failed (node:12tuEGhEzrpo1eV7v2QatHXHMy7b1S5gcVSymWYWJSoCFUMGnaL, address:77.90.31.15:4716): protocol: write tcp 188.166.225.170:53774->77.90.31.15:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:11.437337201 (505h55m9.988807663s) [noise=true , address=77.90.31.12:4623 , piece_id=KV3HWTH37MNBNUL3QZKBIBWFSKZVTA4OU4CSU2JBFL633DUMR7TQ, seg_pos={0 0}, upload=445495]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: rpc: dial tcp 77.90.31.12:4623: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:11.437366667 (505h55m9.988837118s) [noise=true , address=77.90.31.16:4739 , piece_id=ZCP2QMOZVLVWASVFOI5T4XD5I4I6PTA5GTDTNELUWPOUKNGIV7XA, seg_pos={0 0}, upload=445495]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: write tcp 188.166.225.170:58708->77.90.31.16:4739: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:59:11.437393648 (505h55m9.988864119s) [seg_pos={0 0}, upload=445495]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Nov 30 09:59:11.693679980 (505h55m10.245150431s) [upload=445495]: Done uploading segments... Nov 30 09:59:11.694138644 (505h55m10.245609105s) [upload=445495]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.259432656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.248413053s trace: https://jaeger.dev.storj.io/trace/c121f61fd6d1e4f FAIL (2025-11-30 09:58:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== hRequestItem_ObjectBegin Nov 30 09:57:11.673737240 (505h53m10.225207701s) [upload=445479]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 30 09:57:11.925072844 (505h53m10.476543305s) [seg_pos={0 0}, upload=445479]: Begin upload segment... Nov 30 09:57:11.925144721 (505h53m10.476615182s) [seg_pos={0 0}, upload=445479]: Done begin upload segment. Nov 30 09:57:11.925185048 (505h53m10.476655509s) [noise=true , address=77.90.31.10:4539 , piece_id=IFSA6VMNBPF5LKJLSHU6FCJHBDWGXOX3DGCFOIWC2K2IMUYQ7G6Q, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.925263738 (505h53m10.476734199s) [noise=true , address=64.49.13.169:4714 , piece_id=CX3O3MKDSMNAXULTTES5MTUT3QNPJW3ZVILGMNAWN7BGOBL4ZYZQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.926186154 (505h53m10.477656605s) [noise=true , address=45.91.116.14:4221 , piece_id=ISPYKGNQICID5K2CISXL7ZJ6PMVDAM5PL7ZVOHFGEJKU3NTHEHCA, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.926677800 (505h53m10.478148251s) [noise=true , address=145.79.227.52:4454 , piece_id=AT3WI2VKHAA2XCKJYOCBWOCXFGI5UJ77B6I7Q73BD6BL3UG7IOSQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.926701375 (505h53m10.478171836s) [noise=true , address=77.90.31.21:4889 , piece_id=XIQLKOBPT3V6ACR5H4CI7U2CE3GE7XXQ5B7SFRDPZ6H5J27VFBSA, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.927199764 (505h53m10.478670225s) [noise=true , address=64.49.13.139:4469 , piece_id=CDUDB5UI4XPSMX4F2C7ZISII5X3O6GJNIK4GNW24LAHVDNJOPKPQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.927412229 (505h53m10.47888269s) [noise=true , address=64.49.13.137:4495 , piece_id=5KXKYVAAXLFBLGV6O7NLKKBOT745IEHDGO34PTRLFHZ3E434EUFQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.927996872 (505h53m10.479467333s) [noise=true , address=145.79.227.26:4669 , piece_id=PPPLJHJ6PF6VQUQXRKTV3N5FQG2Z6Q4IG3B54Q6LXTAXRK4AGEFA, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.928113343 (505h53m10.479583805s) [noise=true , address=77.90.31.11:4556 , piece_id=LEOPKWBTOD7GVYYMV3SOVER7VK3UKKSAYDADZFRFWT35GHYKD7DQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.928609509 (505h53m10.48007997s) [noise=true , address=64.49.13.140:4515 , piece_id=PUVOTF6YAHYP3UPJMEDC6BN5YEXMOQ3MLGVBDBL2QMI33RGD4V3Q, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.928952462 (505h53m10.480422913s) [noise=true , address=77.90.31.13:4636 , piece_id=T566EBP6JDQ6SGG3N55KVRS5FDTXLDYBNHA7ENZD4HU436HVRYKQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.929059225 (505h53m10.480529676s) [noise=true , address=145.79.227.20:4443 , piece_id=NBOJRZKQMLGQJRJ5AZ3C3R3ASTBC3UGEAF6ZIGYNEOSQTNUMUASQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.929544419 (505h53m10.48101488s) [noise=true , address=45.91.116.13:4158 , piece_id=Q6PPDH565MRHQ5BP2546FAI3LZSVBXG6VJWPJPA7TZY2DY5A6QSA, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.929860321 (505h53m10.481330812s) [noise=true , address=145.79.227.11:4760 , piece_id=7UXOOCXKZHMABGWJEGP3SR5PBP47242UE5ILFL25SMCAPOEJIC3Q, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.930059660 (505h53m10.481530131s) [noise=true , address=45.91.116.10:4084 , piece_id=DOCL2WCUVQQ4VCDP6TEWVS6TTTUYLD7LRWV22JRNU7DOL6O4QRCA, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.930477395 (505h53m10.481947856s) [noise=true , address=45.91.116.8:4063 , piece_id=L6XMNGOTWBS5P3TP4GPSDMKA3YDDBFTETHUXUJRE3GJSSTXFTU4Q, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.930652529 (505h53m10.48212299s) [noise=true , address=77.90.31.16:4750 , piece_id=XC2RI6KVRAASLUIFPGT5STGX32FH6AZJPZNABOE2J4HRMV7I7RYA, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:11.931356039 (505h53m10.4828265s) [noise=true , address=77.90.31.14:4693 , piece_id=SHNL2ALP6HSQ6273IWGYLV7NPDYXOQGVIWH4KD5MY65Q7O4QNBCQ, seg_pos={0 0}, upload=445479]: Uploading piece... Nov 30 09:57:13.232869575 (505h53m11.784340036s) [upload=445479]: Next returned nil segment Nov 30 09:57:13.232888462 (505h53m11.784358923s) [upload=445479]: Waiting for error group managing segments... Nov 30 09:57:13.561193571 (505h53m12.112664032s) [noise=true , address=64.49.13.139:4469 , piece_id=CDUDB5UI4XPSMX4F2C7ZISII5X3O6GJNIK4GNW24LAHVDNJOPKPQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.638515979 (505h53m12.18998644s) [noise=true , address=64.49.13.137:4495 , piece_id=5KXKYVAAXLFBLGV6O7NLKKBOT745IEHDGO34PTRLFHZ3E434EUFQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.719048439 (505h53m12.2705189s) [noise=true , address=64.49.13.140:4515 , piece_id=PUVOTF6YAHYP3UPJMEDC6BN5YEXMOQ3MLGVBDBL2QMI33RGD4V3Q, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.820710204 (505h53m12.372180665s) [noise=true , address=145.79.227.52:4454 , piece_id=AT3WI2VKHAA2XCKJYOCBWOCXFGI5UJ77B6I7Q73BD6BL3UG7IOSQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.821333081 (505h53m12.372803542s) [noise=true , address=145.79.227.20:4443 , piece_id=NBOJRZKQMLGQJRJ5AZ3C3R3ASTBC3UGEAF6ZIGYNEOSQTNUMUASQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.825642717 (505h53m12.377113168s) [noise=true , address=145.79.227.11:4760 , piece_id=7UXOOCXKZHMABGWJEGP3SR5PBP47242UE5ILFL25SMCAPOEJIC3Q, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.831729247 (505h53m12.383199708s) [noise=true , address=145.79.227.26:4669 , piece_id=PPPLJHJ6PF6VQUQXRKTV3N5FQG2Z6Q4IG3B54Q6LXTAXRK4AGEFA, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.880478005 (505h53m12.431948466s) [noise=true , address=45.91.116.10:4084 , piece_id=DOCL2WCUVQQ4VCDP6TEWVS6TTTUYLD7LRWV22JRNU7DOL6O4QRCA, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.882539089 (505h53m12.43400955s) [noise=true , address=45.91.116.8:4063 , piece_id=L6XMNGOTWBS5P3TP4GPSDMKA3YDDBFTETHUXUJRE3GJSSTXFTU4Q, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.926646008 (505h53m12.478116469s) [noise=true , address=45.91.116.13:4158 , piece_id=Q6PPDH565MRHQ5BP2546FAI3LZSVBXG6VJWPJPA7TZY2DY5A6QSA, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:13.974212485 (505h53m12.525682946s) [noise=true , address=45.91.116.14:4221 , piece_id=ISPYKGNQICID5K2CISXL7ZJ6PMVDAM5PL7ZVOHFGEJKU3NTHEHCA, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:57:14.411970896 (505h53m12.963441347s) [noise=true , address=64.49.13.169:4714 , piece_id=CX3O3MKDSMNAXULTTES5MTUT3QNPJW3ZVILGMNAWN7BGOBL4ZYZQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: <nil> Nov 30 09:58:10.965412549 (505h54m9.51688301s) [noise=true , address=77.90.31.10:4539 , piece_id=IFSA6VMNBPF5LKJLSHU6FCJHBDWGXOX3DGCFOIWC2K2IMUYQ7G6Q, seg_pos={0 0}, upload=445479]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:10.965457364 (505h54m9.516927835s) [noise=true , address=77.90.31.13:4636 , piece_id=T566EBP6JDQ6SGG3N55KVRS5FDTXLDYBNHA7ENZD4HU436HVRYKQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: ecclient: upload failed (node:123r9xrR18b29rsRY6APtib52WagU7ttLA7MHY4EQ7inh6BjdCB, address:77.90.31.13:4636): protocol: rpc: dial tcp 77.90.31.13:4636: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:10.965785108 (505h54m9.517255579s) [noise=true , address=77.90.31.16:4750 , piece_id=XC2RI6KVRAASLUIFPGT5STGX32FH6AZJPZNABOE2J4HRMV7I7RYA, seg_pos={0 0}, upload=445479]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:10.966174179 (505h54m9.51764464s) [noise=true , address=77.90.31.14:4693 , piece_id=SHNL2ALP6HSQ6273IWGYLV7NPDYXOQGVIWH4KD5MY65Q7O4QNBCQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:10.966397073 (505h54m9.517867544s) [noise=true , address=77.90.31.21:4889 , piece_id=XIQLKOBPT3V6ACR5H4CI7U2CE3GE7XXQ5B7SFRDPZ6H5J27VFBSA, seg_pos={0 0}, upload=445479]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:10.966619136 (505h54m9.518089597s) [noise=true , address=77.90.31.11:4556 , piece_id=LEOPKWBTOD7GVYYMV3SOVER7VK3UKKSAYDADZFRFWT35GHYKD7DQ, seg_pos={0 0}, upload=445479]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: write tcp 188.166.225.170:37034->77.90.31.11:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 30 09:58:10.966653802 (505h54m9.518124263s) [seg_pos={0 0}, upload=445479]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 30 09:58:11.211761216 (505h54m9.763231677s) [upload=445479]: Done uploading segments... Nov 30 09:58:11.211808105 (505h54m9.763278566s) [upload=445479]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.248428902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.659249754s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:34:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.659318445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.633823425s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:34:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.633855857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.690734156s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:33:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.690749365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.718636369s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:32:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.718666235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.709417124s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:32:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.709462361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.68288511s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:31:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.682907251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.576643186s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:30:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.57668233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.649610421s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:29:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.649664293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.653994305s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:29:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.654025164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.618966955s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.618994359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.562519187s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:27:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.562551236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.672215571s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:26:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.672233023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.692297323s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:26:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.692329344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.743991722s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:25:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.744100069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.768986474s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:24:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.769015248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.496848597s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:23:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.496877291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.729421517s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:23:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.72946031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.57058127s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:22:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.570611056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.686863444s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:21:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.686892609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.793068903s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:21:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.793303991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.727247079s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:20:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.727278559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.716418493s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:19:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.716446706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.687477305s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:19:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.68749572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.593762342s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:18:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.593791045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.671841456s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:18:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.67186995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.772385186s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:17:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.772424091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.824038977s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:17:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.824074523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.456452802s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:16:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.456475126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.664827092s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:15:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.664855647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.668528063s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:14:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.66856294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.721309542s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:13:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.721341824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.64444644s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.644469844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.76542144s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:11:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.765440966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.708411712s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:10:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.708453811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.639580721s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:09:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.639612001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.743013792s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:08:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.743042757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.558573769s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:07:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.558604717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.630293968s trace: https://jaeger.dev.storj.io/trace/28b214f115423b77 FAIL (2025-11-30 07:07:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.402913290 (502h43m43.954383741s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403416118 (502h43m43.954886579s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.403887445 (502h43m43.955357906s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.404543555 (502h43m43.956014016s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405037645 (502h43m43.956508106s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405550622 (502h43m43.957021083s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.405883507 (502h43m43.957353968s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406397115 (502h43m43.957867576s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.406826061 (502h43m43.958296522s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407403771 (502h43m43.958874222s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.407853858 (502h43m43.959324309s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408295048 (502h43m43.959765499s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.408699278 (502h43m43.960169739s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.409105621 (502h43m43.960576083s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.411134606 (502h43m43.962605057s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:45.412039729 (502h43m43.96351019s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Uploading piece... Nov 30 06:47:46.733132764 (502h43m45.284603235s) [upload=442670]: Next returned nil segment Nov 30 06:47:46.733151259 (502h43m45.28462172s) [upload=442670]: Waiting for error group managing segments... Nov 30 06:47:47.072473948 (502h43m45.623944409s) [noise=true , address=64.49.13.154:4935 , piece_id=DD4CGSNRHT6BU5TLJWEKNLJ6EO3FSQRDWCTNVSEKI5QJG2ATXTTQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.282360303 (502h43m45.833830764s) [noise=true , address=145.79.227.14:4844 , piece_id=V3KKSPN3TNJM7PGMEXAGSNVWO3UDD4IAUTKF5B666IJOVYP4YVQA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.301194157 (502h43m45.852664618s) [noise=true , address=77.90.31.11:4583 , piece_id=BK3JAUGZQBOUGPSCTMW7OG6IF6ZUBMIEE5MTFJOWS3LZ6X2QX6QQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.312131063 (502h43m45.863601524s) [noise=true , address=145.79.227.17:4998 , piece_id=2QFE355N6H5CTF467DZ25NKMCMX2URD33SZG4YMUHF2KSIUFCAAQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.322047256 (502h43m45.873517717s) [noise=true , address=77.90.31.10:4539 , piece_id=L523SLJTJFAVPZGRKJTKMS35KFMRFKYIARZAMO7JORTUFZSI3QBQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.331709534 (502h43m45.883179995s) [noise=true , address=77.90.31.16:4745 , piece_id=RY24ZXN2I52GSIOMWMXYKE3J4E63BUF6VUM6OHOAA4RIYDFVRTGA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.334351865 (502h43m45.885822327s) [noise=true , address=77.90.31.17:4793 , piece_id=HSDFWCIZ6B72BGLC4BQZJXLKWWKBXAEFGMJBRVSZHOIXYH6MBPUQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.337367468 (502h43m45.888837929s) [noise=true , address=77.90.31.15:4701 , piece_id=GISXVLZLBW6HWDQ2SQY5UOGZCROUJW5WPXDR2VJKM27N5UYVMHWQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.344891755 (502h43m45.896362227s) [noise=true , address=77.90.31.21:4903 , piece_id=BFEAO6UAACNRWDV5RQNIJYYNF53E25KHZQO57NLHZ6OWWXP6W6YQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.354485234 (502h43m45.905955695s) [noise=true , address=45.91.116.14:4203 , piece_id=WJIWHXMLWNK7Z2M3NQMCKWAZWCSYX7IRLZTAAJ5SQMCESPMG42RA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.363177555 (502h43m45.914648017s) [noise=true , address=77.90.31.18:4821 , piece_id=2HMI3HKYR76PLHQAD7ZWBUDFKXCUOZGAVP7VAOUSZQR2GZ4SCXVA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.395586539 (502h43m45.947057s) [noise=true , address=45.91.116.5:4300 , piece_id=77W6LPIWIWZXYISAIKN2EDHUVBQQXVGCTCL3GV7GF2TUIMUHQYIA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.397780627 (502h43m45.949251088s) [noise=true , address=45.91.116.12:4259 , piece_id=3IYOR4QDCFH6BLFZCGHVTCHMFGCITN7BCV6ACKIR76ZDA67WZMFQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405048085 (502h43m45.956518577s) [noise=true , address=45.91.116.13:4186 , piece_id=BU6RHXLFDGVRHRCWXTYKK52SGJTSCTWWXLHIVZWLZYFJUY3TON5Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: <nil> Nov 30 06:47:47.405079676 (502h43m45.956550127s) [seg_pos={0 0}, upload=442670]: Segment reached optimal threshold of 14 pieces. Nov 30 06:47:47.405376721 (502h43m45.956847182s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405387472 (502h43m45.956857933s) [noise=true , address=45.91.116.7:4035 , piece_id=Z5HOMMHMDWZBUOKVIAJKW3NVMXKASC5A3AGAJFO46DUFT5NFHPKQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405447586 (502h43m45.956918037s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405468997 (502h43m45.956939458s) [noise=true , address=77.90.31.13:4643 , piece_id=FS3QNTULLAZBNIDP5Z3LFYZRE2PBEQLBBI4OI53OORPL2K5BHELQ, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405658237 (502h43m45.957128688s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405664610 (502h43m45.957135061s) [noise=true , address=45.91.116.6:4984 , piece_id=NOPGTIS4E2UCGRI7L2IG2MN4SP3QT3DMAGZT4C6NVNX5VBRGST2Q, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405888796 (502h43m45.957359257s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: expected piece hash; context canceled; EOF Nov 30 06:47:47.405898344 (502h43m45.957368795s) [noise=true , address=64.49.13.169:4715 , piece_id=AXLC2DPJZ2ECUK6FTH7Z4UDMBPDGLVUU5IFKPQHGBEK5BLHUJ3QA, seg_pos={0 0}, upload=442670]: Long tail cancellation due to optimal threshold Nov 30 06:47:47.405920366 (502h43m45.957390827s) [seg_pos={0 0}, upload=442670]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 30 06:47:47.406003544 (502h43m45.957473995s) [upload=442670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 30 06:47:47.406009336 (502h43m45.957479797s) [upload=442670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 30 06:50:11.908963016 (502h46m10.460433487s) [upload=442670]: Done uploading segments... Nov 30 06:50:11.909048348 (502h46m10.460518819s) [upload=442670]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.630320399s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)