Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X.......X.] duration: 4.126148955s trace: https://jaeger.dev.storj.io/trace/c45ec0b274ffa13 FAIL (2025-12-11 00:33:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.126167111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 6.151331564s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:33:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.151360908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................X.......X] duration: 5.000687811s trace: https://jaeger.dev.storj.io/trace/c45ec0b274ffa13 FAIL (2025-12-11 00:32:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000722937s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.97462266s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:32:30 UTC) orbiter warning: context deadline exceeded Duration: 3.974659138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 6.069976477s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.070012317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 4.065142513s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:31:16 UTC) orbiter warning: context deadline exceeded Duration: 4.065169274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 6.668349015s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:31:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.668379945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 5.940315286s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:30:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.940346966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 4.085629002s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:30:24 UTC) orbiter warning: context deadline exceeded Duration: 4.085658568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXXXX.....................................................] duration: 5.86385765s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:29:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.863884352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.XXXXXX....................................................] duration: 6.062740372s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:29:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.062767173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 4.046009382s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:29:21 UTC) orbiter warning: context deadline exceeded Duration: 4.046038958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 4.122811439s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:28:34 UTC) orbiter warning: context deadline exceeded Duration: 4.1228535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.XXXXXX...................................................] duration: 5.751520993s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.751551251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.942792344s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:28:07 UTC) orbiter warning: context deadline exceeded Duration: 3.942946276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 4.494452867s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:27:45 UTC) orbiter warning: context deadline exceeded Duration: 4.494493113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX.XXXXXX..................................................] duration: 6.900197934s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:27:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.900220828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001570308s trace: https://jaeger.dev.storj.io/trace/50ac2e14748bf235 FAIL (2025-12-11 00:27:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001589044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXX.XXXXXX.................................................] duration: 5.931250885s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:26:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.931288587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXX.XXXXXX................................................] duration: 5.957990563s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:25:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.958021912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXX.XXXXXX...............................................] duration: 5.650081824s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:24:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.650107843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXX..............................................] duration: 5.918824281s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:23:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.918846914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXXXXXX.XXXXXX.............................................] duration: 5.882043222s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:23:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.882073749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXXXXXX.XXXXXX............................................] duration: 5.793364253s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:22:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.79338927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XXXXXXX.XXXXXX...........................................] duration: 5.923886283s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:21:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.923914348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X..XXXXXXX.XXXXXX..........................................] duration: 5.987991141s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:21:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.988024134s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X..XXXXXXX.XXXXXX.........................................] duration: 5.826256011s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:20:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.826285337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.X..XXXXXXX.XXXXXX........................................] duration: 5.908067011s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:20:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.908099102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX.X..XXXXXXX.XXXXXX.......................................] duration: 5.790477485s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:19:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.790509046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXX.X..XXXXXXX.XXXXXX......................................] duration: 5.700590805s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:18:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.700617796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXX.X..XXXXXXX.XXXXXX.....................................] duration: 5.898957407s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:18:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.89905803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXXXX.X..XXXXXXX.XXXXXX....................................] duration: 5.832539776s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:17:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.832564984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XXXXX.X..XXXXXXX.XXXXXX...................................] duration: 5.877532159s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:17:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.877561875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.XXXXX.X..XXXXXXX.XXXXXX..................................] duration: 5.947308815s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:15:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.94734287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX.XXXXX.X..XXXXXXX.XXXXXX.................................] duration: 5.957847061s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:15:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.957884102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXX.XXXXX.X..XXXXXXX.XXXXXX................................] duration: 5.834928855s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.834955846s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.924641221s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:13:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.924662392s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXX.XXXXX.X..XXXXXXX.XXXXXX...............................] duration: 5.777001628s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:13:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.777030524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.172028325s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:12:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.172079683s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXXXX.XXXXX.X..XXXXXXX.XXXXXX..............................] duration: 5.791628733s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.791657899s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.011450374s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:11:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.011474289s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.137528236s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:11:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.137692087s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXXXX.XXXXX.X..XXXXXXX.XXXXXX.............................] duration: 5.761284951s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:11:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.761320589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.752792321s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:11:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.752887713s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.363891753s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:10:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.363912943s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.77447185s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:10:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774500975s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XXXXX.XXXXX.X..XXXXXXX.XXXXXX............................] duration: 5.950357327s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:10:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.950392033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.847967554s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:09:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.847997863s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XXXXX.XXXXX.X..XXXXXXX.XXXXXX...........................] duration: 5.749799017s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:09:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.74984156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XXXXX.XXXXX.X..XXXXXXX.XXXXXX..........................] duration: 5.820841573s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:08:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.820905073s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)