Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.034250064s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:08:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.034277396s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [......XXXXX.XXXXX.X..XXXXXXX.XXXXXX.........................] duration: 6.220317436s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.220343036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.614417943s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:08:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.614444392s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......XXXXX.XXXXX.X..XXXXXXX.XXXXXX........................] duration: 5.778962486s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:07:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.77899078s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.781363179s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:07:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.781383536s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [........XXXXX.XXXXX.X..XXXXXXX.XXXXXX.......................] duration: 5.879746646s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:07:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.879764751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.........XXXXX.XXXXX.X..XXXXXXX.XXXXXX......................] duration: 5.953914763s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:07:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.953942626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.705217408s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:07:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.705245432s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [..........XXXXX.XXXXX.X..XXXXXXX.XXXXXX.....................] duration: 5.977794732s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:06:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.977818947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.061762149s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:06:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.06178923s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.991667808s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:05:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.991723132s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [...........XXXXX.XXXXX.X..XXXXXXX.XXXXXX....................] duration: 5.886532988s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:05:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.886560631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.93450195s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:04:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.934524141s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [............XXXXX.XXXXX.X..XXXXXXX.XXXXXX...................] duration: 6.001212886s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:04:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.001243203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.291016793s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:04:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.291069643s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.............XXXXX.XXXXX.X..XXXXXXX.XXXXXX..................] duration: 5.718986688s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:03:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.719007227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.874741276s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:03:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.874805869s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 4.096558431s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-11 00:03:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.096587977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.030329437s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:03:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.030361229s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [..............XXXXX.XXXXX.X..XXXXXXX.XXXXXX.................] duration: 5.787344593s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:02:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.787375723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.949422198s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:02:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.949445833s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 4.231382364s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-11 00:02:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.231442378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [...............XXXXX.XXXXX.X..XXXXXXX.XXXXXX................] duration: 5.887091308s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:02:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.887126936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.121114953s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:01:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.121289165s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 4.175158845s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-11 00:01:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.175187901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [................XXXXX.XXXXX.X..XXXXXXX.XXXXXX...............] duration: 5.792632198s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:01:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.792663539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.053037864s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-11 00:00:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.053092949s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.................XXXXX.XXXXX.X..XXXXXXX.XXXXXX..............] duration: 5.822829778s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:00:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.822863403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.088447416s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-11 00:00:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.088479488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [..................XXXXX.XXXXX.X..XXXXXXX.XXXXXX.............] duration: 6.047868577s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-11 00:00:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.047902221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.905387723s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:59:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.905420616s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 4.394484763s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:59:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.394505832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.964971451s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:59:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.965005256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [...................XXXXX.XXXXX.X..XXXXXXX.XXXXXX............] duration: 5.832407508s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:59:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.832430862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.621447185s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:59:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.621475968s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.998566818s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:58:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.998605533s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.035604027s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:58:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.035633273s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................XXXXX.XXXXX.X..XXXXXXX.XXXXXX...........] duration: 5.902436712s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:58:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.902463804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.733504898s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:57:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.733536689s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....................XXXXX.XXXXX.X..XXXXXXX.XXXXXX..........] duration: 5.828410049s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:57:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.828433203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.085448578s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:57:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.08547625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.097902965s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:57:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.097933272s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [......................XXXXX.XXXXX.X..XXXXXXX.XXXXXX.........] duration: 5.878891498s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:56:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.878919322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 4.170785057s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:56:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.170800726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.884662697s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:56:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.884688405s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.018200306s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:56:29 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.018219804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......................XXXXX.XXXXX.X..XXXXXXX.XXXXXX........] duration: 5.815114718s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:56:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.81514144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.921210397s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:55:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.92122818s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 4.18941358s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:55:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.18942991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [........................XXXXX.XXXXX.X..XXXXXXX.XXXXXX.......] duration: 5.665061429s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:55:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.665086478s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)