Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.777622622s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:55:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.777806101s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 4.024547877s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:55:00 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.024576051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.........................XXXXX.XXXXX.X..XXXXXXX.XXXXXX......] duration: 5.841787762s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:54:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.841816908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.895660042s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:54:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.895690229s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 4.162825713s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:54:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.16284966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.735719783s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:53:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.73582298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [..........................XXXXX.XXXXX.X..XXXXXXX.XXXXXX.....] duration: 5.719339182s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:53:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.719365922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [...........................XXXXX.XXXXX.X..XXXXXXX.XXXXXX....] duration: 5.639225647s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:53:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.639253018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.903812835s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:53:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.903839266s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 4.095648134s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:53:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.095699291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [............................XXXXX.XXXXX.X..XXXXXXX.XXXXXX...] duration: 5.818024172s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.818039602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.741469894s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:52:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.741499481s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 4.260444927s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:51:57 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.260557162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.............................XXXXX.XXXXX.X..XXXXXXX.XXXXXX..] duration: 5.661251224s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:51:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.661285812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.91331633s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:51:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.913333833s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 4.197421982s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:51:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.197456739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.986720902s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:51:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.986747743s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [..............................XXXXX.XXXXX.X..XXXXXXX.XXXXXX.] duration: 5.850256897s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:50:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.850274821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [...............................XXXXX.XXXXX.X..XXXXXXX.XXXXXX] duration: 1m0.249063723s trace: https://jaeger.dev.storj.io/trace/540aba6d355f251f FAIL (2025-12-10 23:50:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.476642365 (128h20m41.3477254s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477225115 (128h20m41.34830816s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477274759 (128h20m41.348357794s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477629304 (128h20m41.348712339s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.477893277 (128h20m41.348976322s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478023436 (128h20m41.349106471s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478199561 (128h20m41.349282596s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478459886 (128h20m41.349542931s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478625812 (128h20m41.349708857s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478796006 (128h20m41.349879041s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.478924652 (128h20m41.350007687s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479134220 (128h20m41.350217255s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479248228 (128h20m41.350331273s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479475931 (128h20m41.350558966s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479563277 (128h20m41.350646312s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479833733 (128h20m41.350916778s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:41.479869150 (128h20m41.350952195s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Uploading piece... Dec 10 23:49:42.928001412 (128h20m42.799084477s) [upload=112082]: Next returned nil segment Dec 10 23:49:42.928047029 (128h20m42.799130074s) [upload=112082]: Waiting for error group managing segments... Dec 10 23:49:43.333326095 (128h20m43.20440914s) [noise=true , address=145.79.227.27:4699 , piece_id=WQAFO7VA57W2OH4RILBUA4LGXIBMTBELUSXUTX7BKZHPDFEHTSSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.333958269 (128h20m43.205041314s) [noise=true , address=145.79.227.41:4117 , piece_id=GIZ5TM4PAS7QBRPKJ6FTT4AWWZB4ESK5BPTXSNU5WLAJJDRTD6HA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.338230374 (128h20m43.209313419s) [noise=true , address=145.79.227.36:4535 , piece_id=5JFYOPI5MQWFCXAKMJILTTGKPNLTZ2PPO6IBSY4YUJFUHZSQPDLQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.339875867 (128h20m43.210958912s) [noise=true , address=145.79.227.31:4804 , piece_id=EDVYKAKIV6EKLH3HUNBSWJJQ3VF3CCMTXD3DDIIKEF45T3RTEUQA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.345499695 (128h20m43.21658273s) [noise=true , address=145.79.227.52:4464 , piece_id=BV6ATCYL3JNEKWK7XPZCEEZEQ2SNZAPXL3SOK6LIFTFDXB4XP24A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.365093675 (128h20m43.23617671s) [noise=true , address=145.79.227.45:4263 , piece_id=3PV5BSNSZ3PZCKNGFWNJLDXFYQQLEZIJTHUOS7AH7WGPT366HXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.462600440 (128h20m43.333683485s) [noise=true , address=45.91.116.8:4062 , piece_id=UWLLNOFKIATC6JR5TY6US3EKAMRZREY4O4QBS7B6A4OMG7BYERCA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.505189357 (128h20m43.376272402s) [noise=true , address=45.91.116.14:4202 , piece_id=NYPKZKHJL7HZC7YBXLBJME4I7RGSIIWMQQUVVEEOA7VN36FFZ26A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.539483797 (128h20m43.410566842s) [noise=true , address=45.91.116.7:4029 , piece_id=T24V4RBUPPT7HJUEYK4URTWJBLPROH7XTM3UCKSCT2CTCCD7RXQQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:49:43.559671859 (128h20m43.430754904s) [noise=true , address=45.91.116.10:4115 , piece_id=POLHXHSUNYKEJALGJZB7QZATQKJ2EYXIGAABQWMEK5VJTVIAEUGQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: <nil> Dec 10 23:50:40.536371685 (128h21m40.40745472s) [noise=true , address=77.90.31.17:4798 , piece_id=DCI2R4YBFDAG766SQJXVCC4WQNVB5KX5IGMHX27PFSEOAFP5ST6A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536514757 (128h21m40.407597813s) [noise=true , address=77.90.31.21:4896 , piece_id=G4XISOCMU4CMQHM455STAPJSYYQS5FYCIV2NMFLVKXMHW3UPBOWA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536589009 (128h21m40.407672054s) [noise=true , address=77.90.31.16:4749 , piece_id=4PATOMQIBFBJHSTUBNPVWGF3OM6VUWYGYV5XRWADTQ7MHFQJ5DSQ, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536824116 (128h21m40.407907161s) [noise=true , address=77.90.31.19:4858 , piece_id=H5EZD7RNVZUHKERKPJRUOGN3ZIFAUWRV74GJMCL4KTBI2OEJ6GTA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, address:77.90.31.19:4858): protocol: rpc: dial tcp 77.90.31.19:4858: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.536867439 (128h21m40.407950495s) [noise=true , address=77.90.31.13:4646 , piece_id=6ODRF3FTP5CNXB6U4YAK64FYXOLBRRA2P2NZ7UKOVGZ2XIT6EC2A, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: rpc: dial tcp 77.90.31.13:4646: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537018397 (128h21m40.408101442s) [noise=true , address=77.90.31.15:4715 , piece_id=J4KJFMBNPPBV5ZEMY6LFA2FVWFWJC42R3CWADC43NSTWIP7KLGDA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: write tcp 188.166.225.170:56526->77.90.31.15:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537147563 (128h21m40.408230598s) [noise=true , address=77.90.31.10:4532 , piece_id=6A4QYDLOUIFPUFU47AXCNOYEF7H4E2B4SDDPXZ2MVARO4ENXTKHA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:45930->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537419129 (128h21m40.408502184s) [noise=true , address=77.90.31.14:4681 , piece_id=67WO7A5C6X26475DH7P46AVSRLUB2EHCQM5GLB6VFWM5L3RHCSZA, seg_pos={0 0}, upload=112082]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:40.537486307 (128h21m40.408569352s) [seg_pos={0 0}, upload=112082]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:40.782729683 (128h21m40.653812718s) [upload=112082]: Done uploading segments... Dec 10 23:50:40.782769639 (128h21m40.653852684s) [upload=112082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.249075708s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.897883063s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:50:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.897907701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.768088746s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:50:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.768120004s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.674682799s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:49:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.674707687s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [................................XXXXX.XXXXX.X..XXXXXXX.XXXXX] duration: 1m0.250646567s trace: https://jaeger.dev.storj.io/trace/2893dacec9b261e1 FAIL (2025-12-10 23:49:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== OPML2MAUA, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.001833455 (128h19m40.8729165s) [noise=true , address=145.79.227.21:4503 , piece_id=K7RHT57AHUXDJPCVHQHJP57J2CVFYE444BZ3GTLDZSLXVU6EKIHQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.002462162 (128h19m40.873545217s) [noise=true , address=145.79.227.12:4793 , piece_id=CXI24HK34XX5GHVKPB5LBAOOISHKEY3X3TWHH7UQVUHY3URZNYHA, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.003039912 (128h19m40.874122957s) [noise=true , address=77.90.31.10:4531 , piece_id=IFNLNCN2I4I4WO7CAEL7KRBCNYHEUGTCOYRPJK772AY5NE6AQNGA, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.003176563 (128h19m40.874259618s) [noise=true , address=77.90.31.21:4906 , piece_id=LZE3HJDYUJAJDIQ5ZG7VBK7UFBKSIHAGEQKZ44JHE3T3W4SK7T5A, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.003533131 (128h19m40.874616166s) [noise=true , address=77.90.31.14:4682 , piece_id=PDLQXM7SRQQGGJDYN342QHD64DGESAR25JIJ6LQMPMBPEW2JZTXQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.003870304 (128h19m40.874953349s) [noise=true , address=145.79.227.32:4859 , piece_id=454IXZ33XIPVX4ITTIEJB7VMVMF5J7YG7PQOUPIRFKWFU2OUBPMA, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.004041710 (128h19m40.875124745s) [noise=true , address=145.79.227.44:4220 , piece_id=3YAZOMXB4DN2RFLG4CQY6U4P4VM3VA2FOGZOOC5VGDOYZA6WZCAQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.004441581 (128h19m40.875524636s) [noise=true , address=5.231.73.4:4005 , piece_id=MZ3QCCLLDYKCMQVUTSLR44UGPJTPONJN3LOTKDDRJ2YCJ62NWYUQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.004526253 (128h19m40.875609298s) [noise=true , address=77.90.31.13:4643 , piece_id=LWI6IQUR7SVBLZTWSPJTGS64F57XEGJMSFOG2FE6C2HI36A2KM2Q, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.004969246 (128h19m40.876052291s) [noise=true , address=5.231.73.5:4031 , piece_id=OLOOZSZLPDJIJDXI75SIHO3BXOYDGPHQOVIJX2IA754FSIE3GOPA, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.005016927 (128h19m40.876099962s) [noise=true , address=45.91.116.14:4208 , piece_id=MDYOFSGU3YDN5COM7RIUBTC3PNPK5SSJFOHM5TXX5J2QU43QF7ZA, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.005404054 (128h19m40.876487099s) [noise=true , address=77.90.31.21:4902 , piece_id=KDH2NZNUI63ZDPB4PFG3VDTEDPBUAZJ42MGFXOSODXGWSZSQYZGQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.005527819 (128h19m40.876610874s) [noise=true , address=145.79.227.16:4930 , piece_id=QLGLL6RP7BHLSF7ALEH422ZUBXES7EJ4VQ3ELSSB66XBLBNCZXMA, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.005754381 (128h19m40.876837416s) [noise=true , address=145.79.227.8:4335 , piece_id=ZDPBIFILQCKQ2FID54HIZJKGD4VZZILYF2USNYWAICLPIRSZTO7A, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.006121860 (128h19m40.877204906s) [noise=true , address=77.90.31.16:4757 , piece_id=F7ZGQFO42AFI7RIVFMHAYDQEHYFOMG65ER4WAYHC75JNOG7SRSBQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.008703778 (128h19m40.879786833s) [noise=true , address=77.90.31.11:4563 , piece_id=QX5LM77YXJMWYPUTUFIFZ2YDCFE7L7WL5WJEYJF52TS6SLNSYFSQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:41.009370156 (128h19m40.880453202s) [noise=true , address=45.91.116.10:4115 , piece_id=Q4W6VJR6SKM4DIF5S7WG6D5EZPEMLGSUP2OYMQYM46KV7PNSMBNQ, seg_pos={0 0}, upload=112069]: Uploading piece... Dec 10 23:48:42.456012962 (128h19m42.327096007s) [upload=112069]: Next returned nil segment Dec 10 23:48:42.456028381 (128h19m42.327111426s) [upload=112069]: Waiting for error group managing segments... Dec 10 23:48:42.840612439 (128h19m42.711695474s) [noise=true , address=5.231.73.5:4031 , piece_id=OLOOZSZLPDJIJDXI75SIHO3BXOYDGPHQOVIJX2IA754FSIE3GOPA, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:42.842527766 (128h19m42.713610811s) [noise=true , address=145.79.227.21:4503 , piece_id=K7RHT57AHUXDJPCVHQHJP57J2CVFYE444BZ3GTLDZSLXVU6EKIHQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:42.844274943 (128h19m42.715357978s) [noise=true , address=5.231.73.4:4005 , piece_id=MZ3QCCLLDYKCMQVUTSLR44UGPJTPONJN3LOTKDDRJ2YCJ62NWYUQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:42.849226522 (128h19m42.720309577s) [noise=true , address=145.79.227.8:4335 , piece_id=ZDPBIFILQCKQ2FID54HIZJKGD4VZZILYF2USNYWAICLPIRSZTO7A, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:42.882542209 (128h19m42.753625244s) [noise=true , address=145.79.227.12:4793 , piece_id=CXI24HK34XX5GHVKPB5LBAOOISHKEY3X3TWHH7UQVUHY3URZNYHA, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:42.920300808 (128h19m42.791383843s) [noise=true , address=145.79.227.32:4859 , piece_id=454IXZ33XIPVX4ITTIEJB7VMVMF5J7YG7PQOUPIRFKWFU2OUBPMA, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:42.974200199 (128h19m42.845283244s) [noise=true , address=45.91.116.10:4115 , piece_id=Q4W6VJR6SKM4DIF5S7WG6D5EZPEMLGSUP2OYMQYM46KV7PNSMBNQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:43.076855799 (128h19m42.947938834s) [noise=true , address=45.91.116.14:4208 , piece_id=MDYOFSGU3YDN5COM7RIUBTC3PNPK5SSJFOHM5TXX5J2QU43QF7ZA, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:43.809095402 (128h19m43.680178437s) [noise=true , address=145.79.227.44:4220 , piece_id=3YAZOMXB4DN2RFLG4CQY6U4P4VM3VA2FOGZOOC5VGDOYZA6WZCAQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:48:43.817441314 (128h19m43.688524359s) [noise=true , address=145.79.227.16:4930 , piece_id=QLGLL6RP7BHLSF7ALEH422ZUBXES7EJ4VQ3ELSSB66XBLBNCZXMA, seg_pos={0 0}, upload=112069]: Done uploading piece. err: <nil> Dec 10 23:49:40.061409158 (128h20m39.932492213s) [noise=true , address=77.90.31.17:4788 , piece_id=66RD33Y7P4P255IXEMZ3PO5HOFUOUTNSO5DFDSKQZIFOPML2MAUA, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:1wsDBstADUY9x7M3jLHm5SMZr6KNYQWQtp8at2oFowUbGw8KDq, address:77.90.31.17:4788): protocol: rpc: dial tcp 77.90.31.17:4788: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.061604129 (128h20m39.932687174s) [noise=true , address=77.90.31.10:4531 , piece_id=IFNLNCN2I4I4WO7CAEL7KRBCNYHEUGTCOYRPJK772AY5NE6AQNGA, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: rpc: dial tcp 77.90.31.10:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.061680956 (128h20m39.932763991s) [noise=true , address=77.90.31.11:4563 , piece_id=QX5LM77YXJMWYPUTUFIFZ2YDCFE7L7WL5WJEYJF52TS6SLNSYFSQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb, address:77.90.31.11:4563): protocol: rpc: dial tcp 77.90.31.11:4563: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.062035571 (128h20m39.933118636s) [noise=true , address=77.90.31.13:4643 , piece_id=LWI6IQUR7SVBLZTWSPJTGS64F57XEGJMSFOG2FE6C2HI36A2KM2Q, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA, address:77.90.31.13:4643): protocol: rpc: dial tcp 77.90.31.13:4643: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.062075988 (128h20m39.933159043s) [noise=true , address=77.90.31.21:4906 , piece_id=LZE3HJDYUJAJDIQ5ZG7VBK7UFBKSIHAGEQKZ44JHE3T3W4SK7T5A, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.062513230 (128h20m39.933596265s) [noise=true , address=77.90.31.16:4757 , piece_id=F7ZGQFO42AFI7RIVFMHAYDQEHYFOMG65ER4WAYHC75JNOG7SRSBQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 188.166.225.170:59202->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.062524251 (128h20m39.933607286s) [noise=true , address=77.90.31.14:4682 , piece_id=PDLQXM7SRQQGGJDYN342QHD64DGESAR25JIJ6LQMPMBPEW2JZTXQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d, address:77.90.31.14:4682): protocol: rpc: dial tcp 77.90.31.14:4682: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.062702931 (128h20m39.933785987s) [noise=true , address=77.90.31.21:4902 , piece_id=KDH2NZNUI63ZDPB4PFG3VDTEDPBUAZJ42MGFXOSODXGWSZSQYZGQ, seg_pos={0 0}, upload=112069]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:44276->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:40.062732738 (128h20m39.933815783s) [seg_pos={0 0}, upload=112069]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:49:40.310895925 (128h20m40.18197897s) [upload=112069]: Done uploading segments... Dec 10 23:49:40.310928547 (128h20m40.182011582s) [upload=112069]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.250657777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 4.129076773s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:49:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.129201731s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.652116082s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:48:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.652139648s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.965448715s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:48:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.965468402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [.................................XXXXX.XXXXX.X..XXXXXXX.XXXX] duration: 1m0.253542148s trace: https://jaeger.dev.storj.io/trace/738aedef283234cf FAIL (2025-12-10 23:48:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ad=112053]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:47:40.277488141 (128h18m40.148571176s) [upload=112053]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:47:40.528807428 (128h18m40.399890473s) [seg_pos={0 0}, upload=112053]: Begin upload segment... Dec 10 23:47:40.528863825 (128h18m40.39994687s) [seg_pos={0 0}, upload=112053]: Done begin upload segment. Dec 10 23:47:40.528896707 (128h18m40.399979742s) [noise=true , address=45.91.116.14:4200 , piece_id=XD42XICIEMP56VRRCFCN5XFO6BKDJUUHSXSBL56G37HXMUT6VEIQ, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.530047428 (128h18m40.401130473s) [noise=true , address=145.79.227.31:4804 , piece_id=BCIYOUFQG5TUOSDE3MZIRNPNEN74OPYGECSDAABB3RLHUHHB255A, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.530453642 (128h18m40.401536677s) [noise=true , address=77.90.31.12:4610 , piece_id=XI6FKL3FI4RLUQ6VEE6ZKHPAZTGM5F5GHTHERQ2UR6PX2DNWXX4Q, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.530762010 (128h18m40.401845045s) [noise=true , address=77.90.31.14:4693 , piece_id=7OBKSEKIL6Z2Y7V6TPFKLW3FEKSZOWYL2S6XY5D5M4BKU3NEJXAA, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.531042554 (128h18m40.402125589s) [noise=true , address=145.79.227.33:4892 , piece_id=6KI3D2PRNLMHY74I6RWU3YK4T3GVIIYG4CTBWKCIFGXAH5WLZE4Q, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.531355229 (128h18m40.402438274s) [noise=true , address=45.91.116.13:4181 , piece_id=5Q6MVZ3LCK62L3YEZQZUHPYMBJCXNK54A3WHYBLDEWW7VPWNEYYA, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.531688694 (128h18m40.402771729s) [noise=true , address=77.90.31.13:4656 , piece_id=6KB3YSBOYVS5NYZWZWQLW7HXHXRKFMKKMPZITTLTOJXYTMKSHTDA, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.531975831 (128h18m40.403058876s) [noise=true , address=145.79.227.39:4041 , piece_id=ISGOXOPSSSXATLHKMVRU5VLDBUWCKNYEODF6ZTCI4DDPTBEJD63Q, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.532288206 (128h18m40.403371251s) [noise=true , address=5.231.73.4:4022 , piece_id=PSOOQWKRX472ZP574SVRKJ3QRLOPT2M4O5G4RZFHZQYS453KFQ3A, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.532643733 (128h18m40.403726768s) [noise=true , address=45.91.116.5:4331 , piece_id=B24Y2FKPDHZFOSC5SX2KOTP2I2VWROETDACHEPXTGOWBVZFFQYUQ, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.533133245 (128h18m40.40421628s) [noise=true , address=77.90.31.21:4895 , piece_id=NDMTTROW7OYSZYZWIT7MON4R5YWOXBG2BDL3NVAEFO2Q6BKRNW2Q, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.533345950 (128h18m40.404428995s) [noise=true , address=77.90.31.17:4803 , piece_id=5LQC7UJ2MHAGWW6Y7KOXKB3DCBJQEYYBNKYHEMWJGQQ7XV7N26IA, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.533969267 (128h18m40.405052302s) [noise=true , address=145.79.227.52:4477 , piece_id=I6EKJEIUALPWZ3EGMXKYR4NO6SCJCGOA7HKTGCPN76T5UKR4JUDQ, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.534562566 (128h18m40.405645611s) [noise=true , address=45.91.116.11:4131 , piece_id=DV4IIYJHBNXJLUQ6J7UAPHEB44MGI2CLYYJNMI7WBZKPIC365PLA, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.535101462 (128h18m40.406184507s) [noise=true , address=77.90.31.16:4739 , piece_id=2JWW5T5AIIHV5J6WQ2RP5DLKBPXN25ACKIOSXNKKSPEE6Y3GQUSQ, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.536663036 (128h18m40.407746081s) [noise=true , address=145.79.227.15:4899 , piece_id=CQ7GLZOLAIWSXM2BIZ73O626ZK2CPHY6BDLZROLY6TDBASZVRFIQ, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.538264275 (128h18m40.40934731s) [noise=true , address=5.231.73.3:4004 , piece_id=YHIHQOZGSFTK7IEXSK3ITGQWNXGQJL2V4PIY2WSFVREBCHOKHEVA, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:40.539867077 (128h18m40.410950112s) [noise=true , address=45.91.116.6:4988 , piece_id=XMLLSR5FEINHMS32XJHDQGFHBIG7O2AEH3JM5CMBU5N3BCGFDBUQ, seg_pos={0 0}, upload=112053]: Uploading piece... Dec 10 23:47:42.001572382 (128h18m41.872655437s) [upload=112053]: Next returned nil segment Dec 10 23:47:42.001591868 (128h18m41.872674904s) [upload=112053]: Waiting for error group managing segments... Dec 10 23:47:42.390657002 (128h18m42.261740037s) [noise=true , address=145.79.227.33:4892 , piece_id=6KI3D2PRNLMHY74I6RWU3YK4T3GVIIYG4CTBWKCIFGXAH5WLZE4Q, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.406216429 (128h18m42.277299474s) [noise=true , address=5.231.73.3:4004 , piece_id=YHIHQOZGSFTK7IEXSK3ITGQWNXGQJL2V4PIY2WSFVREBCHOKHEVA, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.408193253 (128h18m42.279276288s) [noise=true , address=145.79.227.31:4804 , piece_id=BCIYOUFQG5TUOSDE3MZIRNPNEN74OPYGECSDAABB3RLHUHHB255A, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.411265784 (128h18m42.282348829s) [noise=true , address=145.79.227.39:4041 , piece_id=ISGOXOPSSSXATLHKMVRU5VLDBUWCKNYEODF6ZTCI4DDPTBEJD63Q, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.424745581 (128h18m42.295828626s) [noise=true , address=145.79.227.52:4477 , piece_id=I6EKJEIUALPWZ3EGMXKYR4NO6SCJCGOA7HKTGCPN76T5UKR4JUDQ, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.428544645 (128h18m42.29962768s) [noise=true , address=5.231.73.4:4022 , piece_id=PSOOQWKRX472ZP574SVRKJ3QRLOPT2M4O5G4RZFHZQYS453KFQ3A, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.451993495 (128h18m42.32307654s) [noise=true , address=145.79.227.15:4899 , piece_id=CQ7GLZOLAIWSXM2BIZ73O626ZK2CPHY6BDLZROLY6TDBASZVRFIQ, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.528916778 (128h18m42.399999813s) [noise=true , address=45.91.116.5:4331 , piece_id=B24Y2FKPDHZFOSC5SX2KOTP2I2VWROETDACHEPXTGOWBVZFFQYUQ, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.537843355 (128h18m42.40892639s) [noise=true , address=45.91.116.14:4200 , piece_id=XD42XICIEMP56VRRCFCN5XFO6BKDJUUHSXSBL56G37HXMUT6VEIQ, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.553095316 (128h18m42.424178351s) [noise=true , address=45.91.116.11:4131 , piece_id=DV4IIYJHBNXJLUQ6J7UAPHEB44MGI2CLYYJNMI7WBZKPIC365PLA, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.561772659 (128h18m42.432855704s) [noise=true , address=45.91.116.13:4181 , piece_id=5Q6MVZ3LCK62L3YEZQZUHPYMBJCXNK54A3WHYBLDEWW7VPWNEYYA, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:47:42.604402273 (128h18m42.475485308s) [noise=true , address=45.91.116.6:4988 , piece_id=XMLLSR5FEINHMS32XJHDQGFHBIG7O2AEH3JM5CMBU5N3BCGFDBUQ, seg_pos={0 0}, upload=112053]: Done uploading piece. err: <nil> Dec 10 23:48:39.580003745 (128h19m39.45108678s) [noise=true , address=77.90.31.14:4693 , piece_id=7OBKSEKIL6Z2Y7V6TPFKLW3FEKSZOWYL2S6XY5D5M4BKU3NEJXAA, seg_pos={0 0}, upload=112053]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:39.580289629 (128h19m39.451372674s) [noise=true , address=77.90.31.12:4610 , piece_id=XI6FKL3FI4RLUQ6VEE6ZKHPAZTGM5F5GHTHERQ2UR6PX2DNWXX4Q, seg_pos={0 0}, upload=112053]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:39.580385111 (128h19m39.451468146s) [noise=true , address=77.90.31.17:4803 , piece_id=5LQC7UJ2MHAGWW6Y7KOXKB3DCBJQEYYBNKYHEMWJGQQ7XV7N26IA, seg_pos={0 0}, upload=112053]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: rpc: dial tcp 77.90.31.17:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:39.580601403 (128h19m39.451684438s) [noise=true , address=77.90.31.16:4739 , piece_id=2JWW5T5AIIHV5J6WQ2RP5DLKBPXN25ACKIOSXNKKSPEE6Y3GQUSQ, seg_pos={0 0}, upload=112053]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:48:39.580718046 (128h19m39.451801091s) [noise=true , address=77.90.31.13:4656 , piece_id=6KB3YSBOYVS5NYZWZWQLW7HXHXRKFMKKMPZITTLTOJXYTMKSHTDA, seg_pos={0 0}, upload=112053]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: write tcp 188.166.225.170:41992->77.90.31.13:4656: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:39.580857862 (128h19m39.451940907s) [noise=true , address=77.90.31.21:4895 , piece_id=NDMTTROW7OYSZYZWIT7MON4R5YWOXBG2BDL3NVAEFO2Q6BKRNW2Q, seg_pos={0 0}, upload=112053]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:48:39.580894872 (128h19m39.451977907s) [seg_pos={0 0}, upload=112053]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:48:39.831580582 (128h19m39.702663617s) [upload=112053]: Done uploading segments... Dec 10 23:48:39.831675192 (128h19m39.702758227s) [upload=112053]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.253557157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.967384887s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:48:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.967419553s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.619223779s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:47:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.619242865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [..................................XXXXX.XXXXX.X..XXXXXXX.XXX] duration: 1m0.243602474s trace: https://jaeger.dev.storj.io/trace/584def59ab992cbf FAIL (2025-12-10 23:47:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== d segment... Dec 10 23:46:40.042714718 (128h17m39.913797763s) [seg_pos={0 0}, upload=112037]: Done begin upload segment. Dec 10 23:46:40.042749855 (128h17m39.9138329s) [noise=true , address=145.79.227.40:4055 , piece_id=UPIQ7GJTX5KDIZKRIUCUXYCWVPZ42G6ZVLBAHSYBYZPMX5I5ZZDA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.042849675 (128h17m39.91393271s) [noise=true , address=45.91.116.14:4219 , piece_id=F6NCQ7OBYLN3F2Z6QH56NXZLUWI62LC2PTJCF2A5YPQX77PYWU3A, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.043392188 (128h17m39.914475223s) [noise=true , address=145.79.227.3:4158 , piece_id=UAXAP3M4QGMKC74ZT2BR7X7EZOAOIISRAE4O2ASYUJ5GGHFBPNCQ, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.043737906 (128h17m39.914820961s) [noise=true , address=145.79.227.7:4321 , piece_id=ZT7XRXSXGJZAWA6XCWWI4K46XF7HJ4SOFJXABPH226FU2HHSVLLA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.044067855 (128h17m39.91515089s) [noise=true , address=145.79.227.27:4699 , piece_id=WA52DHOJ6EFQ4MCIDO6O7MHLHZAMJQKXYCY3IREI4AL2HX7A3XRA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.044523823 (128h17m39.915606858s) [noise=true , address=145.79.227.33:4889 , piece_id=2TSBIUAD2H5AXWJQW6WGE473C7VPRGWPJMHXWEIXFFP3A6A4GPBQ, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.044966837 (128h17m39.916049882s) [noise=true , address=45.91.116.13:4159 , piece_id=2NB3BBVJXVUHCPVESA53DESN62QE5MHHCJO74HP5BEDPTH7KNW7A, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.045401614 (128h17m39.916484659s) [noise=true , address=145.79.227.38:4004 , piece_id=5UZ2DKA4UKBWO6UDLDLWBJIGOT46IGM6VHE327JN4TNQ46AIOJMQ, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.045972862 (128h17m39.917055907s) [noise=true , address=77.90.31.17:4794 , piece_id=5H5H4XCIHHMUJTBL5WEWNX6P2BAW6KDD6WY3RESAUWDI2N2D3ROQ, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.046563496 (128h17m39.917646541s) [noise=true , address=77.90.31.19:4874 , piece_id=DIBC3LT57ABGAKBLCVPXMVD4SPNZHY2JNCLNWY7O4UZUK4D7RNRA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.047036357 (128h17m39.918119402s) [noise=true , address=77.90.31.12:4613 , piece_id=63PLC6UV6EYERK26HDO4EQ4SZVNR3BKCGYTKKZMKGXY4WHJ4AERA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.048040048 (128h17m39.919123094s) [noise=true , address=77.90.31.15:4724 , piece_id=ZELA354PNVQA5372U3XQBH6EXMZQVEAORXPVBKCJ3MJECDO5HHPA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.048665870 (128h17m39.919748905s) [noise=true , address=145.79.227.24:4613 , piece_id=SMZQ7P2FTXCCVVQ2KVXF7BU4FAMRYYWNZVDR337AMVAQAM7ZQMFQ, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.049231998 (128h17m39.920315033s) [noise=true , address=145.79.227.43:4163 , piece_id=4CTFRDBI72LB6LL27L4IBZKZA23XO3TFAQNSRJJIDDRP3FHIBKLA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.050739518 (128h17m39.921822563s) [noise=true , address=77.90.31.18:4827 , piece_id=EH3BNGAGV5YNLUHIL3IA7MDKVAW46HAWPZ6LRCCVVB6NWSABYBFQ, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.052390503 (128h17m39.923473548s) [noise=true , address=77.90.31.13:4635 , piece_id=JJO2753A3UOZG32IQFKVZOOPU3AI4SILFV2IFPQG6547NTDXO2SA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.054831150 (128h17m39.925914185s) [noise=true , address=45.91.116.8:4065 , piece_id=3M6J23OQTBXQ3C2GCHRIUOAX6SPLME6NVD7ACYYR5SATJOSZG6NA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:40.059171123 (128h17m39.930254168s) [noise=true , address=77.90.31.10:4536 , piece_id=EZRYPRWQJJRVHYAOTCGVGV77HLZSOWFRCLU5AAGRGLZOGXGNQJMA, seg_pos={0 0}, upload=112037]: Uploading piece... Dec 10 23:46:41.524916352 (128h17m41.395999397s) [upload=112037]: Next returned nil segment Dec 10 23:46:41.524933995 (128h17m41.39601704s) [upload=112037]: Waiting for error group managing segments... Dec 10 23:46:41.914824850 (128h17m41.785907885s) [noise=true , address=145.79.227.33:4889 , piece_id=2TSBIUAD2H5AXWJQW6WGE473C7VPRGWPJMHXWEIXFFP3A6A4GPBQ, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:41.927087490 (128h17m41.798170535s) [noise=true , address=145.79.227.3:4158 , piece_id=UAXAP3M4QGMKC74ZT2BR7X7EZOAOIISRAE4O2ASYUJ5GGHFBPNCQ, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:41.936742515 (128h17m41.80782555s) [noise=true , address=145.79.227.43:4163 , piece_id=4CTFRDBI72LB6LL27L4IBZKZA23XO3TFAQNSRJJIDDRP3FHIBKLA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:41.938161116 (128h17m41.809244161s) [noise=true , address=145.79.227.38:4004 , piece_id=5UZ2DKA4UKBWO6UDLDLWBJIGOT46IGM6VHE327JN4TNQ46AIOJMQ, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:41.938371337 (128h17m41.809454402s) [noise=true , address=145.79.227.27:4699 , piece_id=WA52DHOJ6EFQ4MCIDO6O7MHLHZAMJQKXYCY3IREI4AL2HX7A3XRA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:41.939814524 (128h17m41.810897559s) [noise=true , address=145.79.227.24:4613 , piece_id=SMZQ7P2FTXCCVVQ2KVXF7BU4FAMRYYWNZVDR337AMVAQAM7ZQMFQ, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:41.940953704 (128h17m41.812036739s) [noise=true , address=145.79.227.7:4321 , piece_id=ZT7XRXSXGJZAWA6XCWWI4K46XF7HJ4SOFJXABPH226FU2HHSVLLA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:41.951632988 (128h17m41.822716033s) [noise=true , address=145.79.227.40:4055 , piece_id=UPIQ7GJTX5KDIZKRIUCUXYCWVPZ42G6ZVLBAHSYBYZPMX5I5ZZDA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:42.046535035 (128h17m41.91761808s) [noise=true , address=45.91.116.13:4159 , piece_id=2NB3BBVJXVUHCPVESA53DESN62QE5MHHCJO74HP5BEDPTH7KNW7A, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:42.072293062 (128h17m41.943376117s) [noise=true , address=45.91.116.8:4065 , piece_id=3M6J23OQTBXQ3C2GCHRIUOAX6SPLME6NVD7ACYYR5SATJOSZG6NA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:46:42.118134931 (128h17m41.989217967s) [noise=true , address=45.91.116.14:4219 , piece_id=F6NCQ7OBYLN3F2Z6QH56NXZLUWI62LC2PTJCF2A5YPQX77PYWU3A, seg_pos={0 0}, upload=112037]: Done uploading piece. err: <nil> Dec 10 23:47:39.109894934 (128h18m38.980977979s) [noise=true , address=77.90.31.10:4536 , piece_id=EZRYPRWQJJRVHYAOTCGVGV77HLZSOWFRCLU5AAGRGLZOGXGNQJMA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:39.110028278 (128h18m38.981111323s) [noise=true , address=77.90.31.15:4724 , piece_id=ZELA354PNVQA5372U3XQBH6EXMZQVEAORXPVBKCJ3MJECDO5HHPA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: ecclient: upload failed (node:12YEcohZBsQSm45orPuT8np4EGgXQaWAM8rCsmUfA1PfNubwLmo, address:77.90.31.15:4724): protocol: rpc: dial tcp 77.90.31.15:4724: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:39.110175418 (128h18m38.981258473s) [noise=true , address=77.90.31.19:4874 , piece_id=DIBC3LT57ABGAKBLCVPXMVD4SPNZHY2JNCLNWY7O4UZUK4D7RNRA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:39.110386029 (128h18m38.981469084s) [noise=true , address=77.90.31.17:4794 , piece_id=5H5H4XCIHHMUJTBL5WEWNX6P2BAW6KDD6WY3RESAUWDI2N2D3ROQ, seg_pos={0 0}, upload=112037]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:39.110685449 (128h18m38.981768494s) [noise=true , address=77.90.31.12:4613 , piece_id=63PLC6UV6EYERK26HDO4EQ4SZVNR3BKCGYTKKZMKGXY4WHJ4AERA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: rpc: dial tcp 77.90.31.12:4613: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:39.110715196 (128h18m38.981798261s) [noise=true , address=77.90.31.18:4827 , piece_id=EH3BNGAGV5YNLUHIL3IA7MDKVAW46HAWPZ6LRCCVVB6NWSABYBFQ, seg_pos={0 0}, upload=112037]: Done uploading piece. err: ecclient: upload failed (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, address:77.90.31.18:4827): protocol: rpc: dial tcp 77.90.31.18:4827: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:39.110868728 (128h18m38.981951764s) [noise=true , address=77.90.31.13:4635 , piece_id=JJO2753A3UOZG32IQFKVZOOPU3AI4SILFV2IFPQG6547NTDXO2SA, seg_pos={0 0}, upload=112037]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: write tcp 188.166.225.170:37744->77.90.31.13:4635: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:47:39.110896420 (128h18m38.981979455s) [seg_pos={0 0}, upload=112037]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:47:39.351929952 (128h18m39.223012997s) [upload=112037]: Done uploading segments... Dec 10 23:47:39.351968675 (128h18m39.22305171s) [upload=112037]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.24362671s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 4.040659588s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:47:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.040686819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 4.04123459s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:46:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.041268936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.818664952s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:46:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.818689629s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [...................................XXXXX.XXXXX.X..XXXXXXX.XX] duration: 1m0.244338812s trace: https://jaeger.dev.storj.io/trace/5d87417faaaac098 FAIL (2025-12-10 23:46:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m25.577807243s) [seg_pos={0 0}, upload=112018]: Done begin upload segment. Dec 10 23:45:25.706779914 (128h16m25.577862949s) [noise=true , address=77.90.31.15:4720 , piece_id=GDL7PHQMKA6BTP75QFRWOFFSE2HLPDWXUAKKWURDC7WAIEZEGIRA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.706799983 (128h16m25.577883018s) [noise=true , address=145.79.227.33:4903 , piece_id=QDT7TCK2X66NIF4YTJM5MXQXUW7MK6VSVR7XOPLX57IDZLL4MQTA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.706897067 (128h16m25.577980112s) [noise=true , address=64.49.13.140:4510 , piece_id=CA4BR37NKJJQWRM7AYWVYWUGFT7PNWQVIFR73UMLIM3N33TPQWLA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.707426516 (128h16m25.578509551s) [noise=true , address=77.90.31.21:4881 , piece_id=UWCTZA36Q7HDTVXK5JMNDA43OJZF7YNEGUMVO7MJGXA4MBZODOQA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.707561682 (128h16m25.578644717s) [noise=true , address=77.90.31.11:4574 , piece_id=BINX2WOOHBWXTDDV7M6SVH7H7KRRU3QNEYW57LXT3IHHCF46VVUA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.707760822 (128h16m25.578843857s) [noise=true , address=77.90.31.18:4830 , piece_id=ZWUAAGJJLM6ZLVPXF4ORXXXJTZTKBEIVKFAHGGSICAUQVZNXB3HQ, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.708126318 (128h16m25.579209353s) [noise=true , address=145.79.227.16:4924 , piece_id=LAYXVBVKUNMZ2QD45VSWHFW2GLOFBGZ7WTFJOA2PBZCMLWQM3LOA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.708459041 (128h16m25.579542076s) [noise=true , address=45.91.116.9:4289 , piece_id=6QV5BRWV7J4IS63OX4KZUU3F3L6HTEW6HU3ZXFDRO5JPSFZZ6LTA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.708736570 (128h16m25.579819615s) [noise=true , address=145.79.227.51:4420 , piece_id=V2H4YKKOR2EHLU4QANWFQD3QNTGHFSVK5XQIDZYS3NU6GEQQ36JQ, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.709020991 (128h16m25.580104036s) [noise=true , address=145.79.227.17:4977 , piece_id=GYVCH5CWWT5WA2V65VSYR5ZVI5S7UXR4NT6JRMNGDDJ32UZOJDOA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.709338807 (128h16m25.580421842s) [noise=true , address=145.79.227.6:4271 , piece_id=3IPZTXIKYH2GMVW2M6EMVZEPIXGAZMSK7QUHQB2G5OLNV4FPDJ3A, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.709690517 (128h16m25.580773552s) [noise=true , address=77.90.31.10:4521 , piece_id=6YIW6ENQU7BKIBMKOB4565MH66CP5BXBTLXMVNM43KNSOX7R4ACA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.710083996 (128h16m25.581167041s) [noise=true , address=77.90.31.12:4614 , piece_id=VHDIUDJPRKLAOFGWI4E4QON5VOREQHYZ36FAC5IPGU2DDGAVWGMA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.710241445 (128h16m25.58132448s) [noise=true , address=145.79.227.42:4154 , piece_id=Q5NFLZUSESI2LL72Q22KALSSUIELAZPYZQH336CASUYSKCQMSXCA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.710770493 (128h16m25.581853538s) [noise=true , address=145.79.227.23:4563 , piece_id=6TST7P64WDHVB6452ICLMFZWDMOGJOYR7GJXPNAELNEZ2SH7ZWLA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.711098377 (128h16m25.582181422s) [noise=true , address=77.90.31.16:4767 , piece_id=V452FNCJLUZJYPTP2QRBFP7TMIATLITCCBNI2C45WCJK6XGVNOXQ, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.711463652 (128h16m25.582546697s) [noise=true , address=45.91.116.14:4205 , piece_id=7K4VXCKFZIRR7D45BDMRZFYGEFSOMMPLHJQKB2YNDT5TNNVDEMRQ, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:25.711694683 (128h16m25.582777718s) [noise=true , address=45.91.116.12:4231 , piece_id=DKDLHYNUDWIHIID2PUPRBU42TFO2LHYXGE7CTAOZNFA7RLYL2ZLA, seg_pos={0 0}, upload=112018]: Uploading piece... Dec 10 23:45:27.062037461 (128h16m26.933120506s) [upload=112018]: Next returned nil segment Dec 10 23:45:27.062074942 (128h16m26.933157987s) [upload=112018]: Waiting for error group managing segments... Dec 10 23:45:27.404923191 (128h16m27.276006226s) [noise=true , address=64.49.13.140:4510 , piece_id=CA4BR37NKJJQWRM7AYWVYWUGFT7PNWQVIFR73UMLIM3N33TPQWLA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.560457861 (128h16m27.431540906s) [noise=true , address=145.79.227.51:4420 , piece_id=V2H4YKKOR2EHLU4QANWFQD3QNTGHFSVK5XQIDZYS3NU6GEQQ36JQ, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.569730487 (128h16m27.440813522s) [noise=true , address=145.79.227.33:4903 , piece_id=QDT7TCK2X66NIF4YTJM5MXQXUW7MK6VSVR7XOPLX57IDZLL4MQTA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.579320478 (128h16m27.450403523s) [noise=true , address=145.79.227.17:4977 , piece_id=GYVCH5CWWT5WA2V65VSYR5ZVI5S7UXR4NT6JRMNGDDJ32UZOJDOA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.585022967 (128h16m27.456106002s) [noise=true , address=145.79.227.23:4563 , piece_id=6TST7P64WDHVB6452ICLMFZWDMOGJOYR7GJXPNAELNEZ2SH7ZWLA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.588544773 (128h16m27.459627808s) [noise=true , address=145.79.227.6:4271 , piece_id=3IPZTXIKYH2GMVW2M6EMVZEPIXGAZMSK7QUHQB2G5OLNV4FPDJ3A, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.592655610 (128h16m27.463738645s) [noise=true , address=145.79.227.42:4154 , piece_id=Q5NFLZUSESI2LL72Q22KALSSUIELAZPYZQH336CASUYSKCQMSXCA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.677700337 (128h16m27.548783372s) [noise=true , address=45.91.116.14:4205 , piece_id=7K4VXCKFZIRR7D45BDMRZFYGEFSOMMPLHJQKB2YNDT5TNNVDEMRQ, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.771078310 (128h16m27.642161345s) [noise=true , address=45.91.116.9:4289 , piece_id=6QV5BRWV7J4IS63OX4KZUU3F3L6HTEW6HU3ZXFDRO5JPSFZZ6LTA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:27.792803399 (128h16m27.663886434s) [noise=true , address=45.91.116.12:4231 , piece_id=DKDLHYNUDWIHIID2PUPRBU42TFO2LHYXGE7CTAOZNFA7RLYL2ZLA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:45:28.525066017 (128h16m28.396149063s) [noise=true , address=145.79.227.16:4924 , piece_id=LAYXVBVKUNMZ2QD45VSWHFW2GLOFBGZ7WTFJOA2PBZCMLWQM3LOA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: <nil> Dec 10 23:46:24.767208660 (128h17m24.638291705s) [noise=true , address=77.90.31.15:4720 , piece_id=GDL7PHQMKA6BTP75QFRWOFFSE2HLPDWXUAKKWURDC7WAIEZEGIRA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 188.166.225.170:57484->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:24.767298602 (128h17m24.638381637s) [noise=true , address=77.90.31.11:4574 , piece_id=BINX2WOOHBWXTDDV7M6SVH7H7KRRU3QNEYW57LXT3IHHCF46VVUA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:24.767701489 (128h17m24.638784554s) [noise=true , address=77.90.31.12:4614 , piece_id=VHDIUDJPRKLAOFGWI4E4QON5VOREQHYZ36FAC5IPGU2DDGAVWGMA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:24.767887093 (128h17m24.638970138s) [noise=true , address=77.90.31.18:4830 , piece_id=ZWUAAGJJLM6ZLVPXF4ORXXXJTZTKBEIVKFAHGGSICAUQVZNXB3HQ, seg_pos={0 0}, upload=112018]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: rpc: dial tcp 77.90.31.18:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:24.768442751 (128h17m24.639525786s) [noise=true , address=77.90.31.21:4881 , piece_id=UWCTZA36Q7HDTVXK5JMNDA43OJZF7YNEGUMVO7MJGXA4MBZODOQA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: ecclient: upload failed (node:12W4YnnqptDeH7t6QVMo988zAyAZ3Kj3dFE1yaxjttuWjrMvKeM, address:77.90.31.21:4881): protocol: rpc: dial tcp 77.90.31.21:4881: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:24.768644013 (128h17m24.639727058s) [noise=true , address=77.90.31.16:4767 , piece_id=V452FNCJLUZJYPTP2QRBFP7TMIATLITCCBNI2C45WCJK6XGVNOXQ, seg_pos={0 0}, upload=112018]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:24.768810480 (128h17m24.639893536s) [noise=true , address=77.90.31.10:4521 , piece_id=6YIW6ENQU7BKIBMKOB4565MH66CP5BXBTLXMVNM43KNSOX7R4ACA, seg_pos={0 0}, upload=112018]: Done uploading piece. err: ecclient: upload failed (node:1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, address:77.90.31.10:4521): protocol: write tcp 188.166.225.170:58870->77.90.31.10:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:46:24.768872399 (128h17m24.639955444s) [seg_pos={0 0}, upload=112018]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:46:25.009520465 (128h17m24.8806035s) [upload=112018]: Done uploading segments... Dec 10 23:46:25.009561533 (128h17m24.880644568s) [upload=112018]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.244353241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 4.254263066s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:46:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.254318412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.651733396s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:45:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.651760788s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [....................................XXXXX.XXXXX.X..XXXXXXX.X] duration: 1m0.240588552s trace: https://jaeger.dev.storj.io/trace/3932bbd4d272c3da FAIL (2025-12-10 23:45:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 02]: Uploading piece... Dec 10 23:44:25.254047572 (128h15m25.125130607s) [noise=true , address=77.90.31.13:4640 , piece_id=S6ZRG65XCLC6P2OB4QK6H7UZSMLWQVHBNYGKPV333TYRXEYW5VKQ, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.254700274 (128h15m25.125783319s) [noise=true , address=5.231.73.3:4029 , piece_id=U3YHKXDUM2PTSS7G57LC63446IUK5FQQDR3NNMIWQXEA2M4FIGLQ, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.254715534 (128h15m25.125798579s) [noise=true , address=77.90.31.17:4804 , piece_id=D5GF6LJ3VGMETMAHMGMCX2JWZX2PVF74F3MSHSF4ACWUKLAGYMWA, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.255121848 (128h15m25.126204893s) [noise=true , address=5.231.73.4:4002 , piece_id=XWDIMGNJQ3EYXCTXJRVLNRKOTB6XMD56FBAEBXQUQVAYW3SY7GTA, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.255306449 (128h15m25.126389494s) [noise=true , address=45.91.116.12:4228 , piece_id=IEBQMVGSLMV3H7N2H7Z735D3343XLK356WAGGAJWXI224G2TXYNA, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.255522420 (128h15m25.126605465s) [noise=true , address=77.90.31.19:4861 , piece_id=2KE26P2ZILIPML7EO5RAG5XSDSS2MIYQ2OSEL4NJ2GHKXCBDXICQ, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.255789128 (128h15m25.126872173s) [noise=true , address=45.91.116.7:4015 , piece_id=3J4PAOHKSCRL5TNFPRPZZZQHFVMUOUQQKJ555RAGXPHEEWOEMB7A, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.255924215 (128h15m25.12700726s) [noise=true , address=77.90.31.14:4668 , piece_id=GX4UZVK32HVY6NFOX65F4RTKE6LSLIBKLZ6HJCM5Y5VDE35IKFBQ, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.256160565 (128h15m25.12724361s) [noise=true , address=145.79.227.24:4590 , piece_id=CBDAC5D34DVCZU6ZAJED42IQF72Z5GBLJEV7Y5LY72GO2XSOHZTQ, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.256324477 (128h15m25.127407522s) [noise=true , address=77.90.31.15:4715 , piece_id=2UMGM57EM3LRBNEZ2BSYVNZPDB7RYRINI4P6THFZ7QG5X3Z3C7ZA, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.256494040 (128h15m25.127577075s) [noise=true , address=145.79.227.33:4874 , piece_id=WBHFHPX2TRNQG7MBGXAFFUA7TS6ZJH7LJZMMXD5DSP33QEEJ2GVQ, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.256759305 (128h15m25.12784235s) [noise=true , address=45.91.116.14:4205 , piece_id=FCP6KRHX3KCYJPBPN6UWYNFRYX4CR5UHINM2KJL54K4TEIG5M55A, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.257195366 (128h15m25.128278411s) [noise=true , address=77.90.31.16:4767 , piece_id=MHD6WCQMCNRNMWD3PDFYDXHTUKVPH57WMPN4LD7TWCWRXIJYHLYA, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.257632157 (128h15m25.128715192s) [noise=true , address=5.231.73.5:4023 , piece_id=ITZJWJ3YI7KN6KBMI24XXLKQUPTUFLYDUP472N2FW5XN4BCA474A, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.258104436 (128h15m25.129187482s) [noise=true , address=145.79.227.30:4782 , piece_id=ZGNFZUTT5Q3SWULTLWDQWIZ4WUNNNQXQS6OJT7MBU3DXEZOY4JHA, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.258641168 (128h15m25.129724213s) [noise=true , address=77.90.31.12:4598 , piece_id=LDQYTALA2IBMNZOWJWWLRUSZEL5NEDVZGFKRKLG3T4CHYOVWVIOQ, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:25.260769210 (128h15m25.131852245s) [noise=true , address=145.79.227.8:4332 , piece_id=RBXODU6HT72WGHWRZR7ZLG3Y3UH4MSTT3GN7V6JSUNXYQG76OE6Q, seg_pos={0 0}, upload=112002]: Uploading piece... Dec 10 23:44:26.720230805 (128h15m26.591313851s) [upload=112002]: Next returned nil segment Dec 10 23:44:26.720263217 (128h15m26.591346262s) [upload=112002]: Waiting for error group managing segments... Dec 10 23:44:27.114707243 (128h15m26.985790298s) [noise=true , address=5.231.73.5:4023 , piece_id=ITZJWJ3YI7KN6KBMI24XXLKQUPTUFLYDUP472N2FW5XN4BCA474A, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.117926733 (128h15m26.989009778s) [noise=true , address=145.79.227.24:4590 , piece_id=CBDAC5D34DVCZU6ZAJED42IQF72Z5GBLJEV7Y5LY72GO2XSOHZTQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.130659848 (128h15m27.001742893s) [noise=true , address=145.79.227.33:4874 , piece_id=WBHFHPX2TRNQG7MBGXAFFUA7TS6ZJH7LJZMMXD5DSP33QEEJ2GVQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.155687163 (128h15m27.026770198s) [noise=true , address=145.79.227.8:4332 , piece_id=RBXODU6HT72WGHWRZR7ZLG3Y3UH4MSTT3GN7V6JSUNXYQG76OE6Q, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.156640730 (128h15m27.027723775s) [noise=true , address=5.231.73.3:4029 , piece_id=U3YHKXDUM2PTSS7G57LC63446IUK5FQQDR3NNMIWQXEA2M4FIGLQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.161672200 (128h15m27.032755235s) [noise=true , address=5.231.73.4:4002 , piece_id=XWDIMGNJQ3EYXCTXJRVLNRKOTB6XMD56FBAEBXQUQVAYW3SY7GTA, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.262556618 (128h15m27.133639673s) [noise=true , address=45.91.116.12:4228 , piece_id=IEBQMVGSLMV3H7N2H7Z735D3343XLK356WAGGAJWXI224G2TXYNA, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.280926918 (128h15m27.152009953s) [noise=true , address=45.91.116.7:4015 , piece_id=3J4PAOHKSCRL5TNFPRPZZZQHFVMUOUQQKJ555RAGXPHEEWOEMB7A, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:27.310192042 (128h15m27.181275087s) [noise=true , address=45.91.116.14:4205 , piece_id=FCP6KRHX3KCYJPBPN6UWYNFRYX4CR5UHINM2KJL54K4TEIG5M55A, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:44:28.140759867 (128h15m28.011842902s) [noise=true , address=145.79.227.30:4782 , piece_id=ZGNFZUTT5Q3SWULTLWDQWIZ4WUNNNQXQS6OJT7MBU3DXEZOY4JHA, seg_pos={0 0}, upload=112002]: Done uploading piece. err: <nil> Dec 10 23:45:24.309956452 (128h16m24.181039487s) [noise=true , address=77.90.31.12:4598 , piece_id=LDQYTALA2IBMNZOWJWWLRUSZEL5NEDVZGFKRKLG3T4CHYOVWVIOQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:12Li7UVDCVmainU4mkxtFyGky2wi6989Qu12BCQGgNn6roNX2A3, address:77.90.31.12:4598): protocol: write tcp 188.166.225.170:48962->77.90.31.12:4598: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.310086960 (128h16m24.181170005s) [noise=true , address=77.90.31.15:4715 , piece_id=2UMGM57EM3LRBNEZ2BSYVNZPDB7RYRINI4P6THFZ7QG5X3Z3C7ZA, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.310204815 (128h16m24.18128786s) [noise=true , address=77.90.31.14:4668 , piece_id=GX4UZVK32HVY6NFOX65F4RTKE6LSLIBKLZ6HJCM5Y5VDE35IKFBQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: rpc: dial tcp 77.90.31.14:4668: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.310515928 (128h16m24.181598973s) [noise=true , address=77.90.31.19:4861 , piece_id=2KE26P2ZILIPML7EO5RAG5XSDSS2MIYQ2OSEL4NJ2GHKXCBDXICQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: rpc: dial tcp 77.90.31.19:4861: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.310652167 (128h16m24.181735202s) [noise=true , address=77.90.31.13:4640 , piece_id=S6ZRG65XCLC6P2OB4QK6H7UZSMLWQVHBNYGKPV333TYRXEYW5VKQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: rpc: dial tcp 77.90.31.13:4640: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.310710658 (128h16m24.181793703s) [noise=true , address=77.90.31.17:4804 , piece_id=D5GF6LJ3VGMETMAHMGMCX2JWZX2PVF74F3MSHSF4ACWUKLAGYMWA, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx, address:77.90.31.17:4804): protocol: write tcp 188.166.225.170:55522->77.90.31.17:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.310998807 (128h16m24.182081842s) [noise=true , address=77.90.31.16:4767 , piece_id=MHD6WCQMCNRNMWD3PDFYDXHTUKVPH57WMPN4LD7TWCWRXIJYHLYA, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.311129485 (128h16m24.18221253s) [noise=true , address=77.90.31.21:4893 , piece_id=IZ7KQQUEEK2A46AZWYXU36JQDGXLLZD6PWRJFZPZUTPJ736V4HDQ, seg_pos={0 0}, upload=112002]: Done uploading piece. err: ecclient: upload failed (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, address:77.90.31.21:4893): protocol: write tcp 188.166.225.170:37818->77.90.31.21:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:24.311180192 (128h16m24.182263238s) [seg_pos={0 0}, upload=112002]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:45:24.548448443 (128h16m24.419531478s) [upload=112002]: Done uploading segments... Dec 10 23:45:24.548488309 (128h16m24.419571354s) [upload=112002]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.240613732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 4.125572182s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:45:00 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.125600455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.641565841s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:44:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.641583394s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.723203536s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:44:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.723229876s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 4.016654009s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:43:53 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.016682043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.631031673s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:43:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.631052541s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 4.171477892s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:43:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.171500004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....................................XXXXX.XXXXX.X..XXXXXXX.] duration: 6.456815106s trace: https://jaeger.dev.storj.io/trace/5336b456b3dc690c FAIL (2025-12-10 23:43:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 583014s) [seg_pos={0 0}, upload=111969]: Begin upload segment... Dec 10 23:42:14.080581605 (128h13m13.95166464s) [seg_pos={0 0}, upload=111969]: Done begin upload segment. Dec 10 23:42:14.080652671 (128h13m13.951735706s) [noise=true , address=145.79.227.30:4778 , piece_id=D6RSHWZZA42HTXEM672GKPC6RP7UWCCHBTEVRLA742737AR5BS4Q, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.080677046 (128h13m13.951760091s) [noise=true , address=45.91.116.12:4240 , piece_id=PYVXH4KTJNZLDVQD4FEOCLP3EGITNB444WJNFTWOVAPJPYQV3BOQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.080763732 (128h13m13.951846777s) [noise=true , address=145.79.227.24:4600 , piece_id=4Y7Z3IIUMKZPDMA5XXU7G4VMQJ2DR2PFDX7POV5JAIPU5NQNEHYQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.081152592 (128h13m13.952235627s) [noise=true , address=45.91.116.13:4184 , piece_id=YGJPB7SIBL2ZQJAWUWLHHZOURQKS2HWTBETMBYLIW3A2HL4MQN3Q, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.081477200 (128h13m13.952560235s) [noise=true , address=45.91.116.5:4321 , piece_id=BLUL5G7Z5EN6UBCXXRBICDST5FHPTEDQB6H3OZ3GD3VHAN3MIZMA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.081850130 (128h13m13.952933166s) [noise=true , address=45.91.116.8:4073 , piece_id=KNND2N5INYU5G4W75AGPOW7M7QPI6TTOZDXT3EHG3NKY5IRXZKSA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.082142687 (128h13m13.953225732s) [noise=true , address=77.90.31.12:4593 , piece_id=76TXIDKPW5M2DJQDLDNDVNP56W3ETFY53YGA6ZX575ONL6A7475Q, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.082667116 (128h13m13.953750151s) [noise=true , address=77.90.31.18:4815 , piece_id=GKZVHO2OWJFF5AYFB76LFED3GLIGOOLJROLBNWINIV3KRXOGK3MA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.083072919 (128h13m13.954155964s) [noise=true , address=45.91.116.9:4291 , piece_id=QT7GRV75S3K4NEEZD6XQ2DVBHVVIF7KP4LISQOOQMGXI4HCIUPRQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.083405002 (128h13m13.954488047s) [noise=true , address=145.79.227.51:4423 , piece_id=J3WR4E6L6S6KGQIQMC6DDDMQJEKSUMXWEBXZPIZ6GIHJNO76Z5KQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.083725111 (128h13m13.954808146s) [noise=true , address=77.90.31.13:4649 , piece_id=BNTF6775HEWBGVIHHUAVD7B73E37VWYVQTVWIK637E4GORYSC52A, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.084217679 (128h13m13.955300724s) [noise=true , address=77.90.31.17:4800 , piece_id=XEXUBXCIZF4VK4Q2ALGTR5QITQOK2DDK27O4GQZF5TSZQHJVPCHQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.084771624 (128h13m13.955854669s) [noise=true , address=77.90.31.16:4766 , piece_id=H74RM2MKTJHFNLEPHWTOZTK34X6LGE7NI5U4NRL3JD5T6BQAVTEQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.086573935 (128h13m13.95765697s) [noise=true , address=77.90.31.12:4603 , piece_id=XCPA4SH4C65HTX7AHTHDEHB2RHJFOKUWFTHYIZ35UUEO2VAYPNEQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.087995993 (128h13m13.959079038s) [noise=true , address=145.79.227.32:4845 , piece_id=GE5B7ZU5VLCRRU7IHY2SIJSD755JFGH6D5TSOAKNDHZSLVQXRMSA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.089335203 (128h13m13.960418248s) [noise=true , address=145.79.227.38:4996 , piece_id=47M5V3FV44NVMRCWKQBUFN7ASJET4BA6I6XYADVLTC75BARTLDBQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.092096240 (128h13m13.963179276s) [noise=true , address=45.91.116.14:4220 , piece_id=HLODTBHS5UASYGFQLWLG2RMDNOB6YQ2GCHXU72MUYXRHWQYWHMWQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.103379665 (128h13m13.97446271s) [noise=true , address=77.90.31.21:4906 , piece_id=PO3CMNXQP4XHEDE47SC276JPPVYR4TDMFFOISIQBSSKOX6GVWXIQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:15.555102291 (128h13m15.426185326s) [upload=111969]: Next returned nil segment Dec 10 23:42:15.555117991 (128h13m15.426201036s) [upload=111969]: Waiting for error group managing segments... Dec 10 23:42:15.955172339 (128h13m15.826255384s) [noise=true , address=145.79.227.24:4600 , piece_id=4Y7Z3IIUMKZPDMA5XXU7G4VMQJ2DR2PFDX7POV5JAIPU5NQNEHYQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:15.962248593 (128h13m15.833331628s) [noise=true , address=145.79.227.32:4845 , piece_id=GE5B7ZU5VLCRRU7IHY2SIJSD755JFGH6D5TSOAKNDHZSLVQXRMSA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:15.981117352 (128h13m15.852200398s) [noise=true , address=145.79.227.51:4423 , piece_id=J3WR4E6L6S6KGQIQMC6DDDMQJEKSUMXWEBXZPIZ6GIHJNO76Z5KQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.010575485 (128h13m15.88165853s) [noise=true , address=145.79.227.38:4996 , piece_id=47M5V3FV44NVMRCWKQBUFN7ASJET4BA6I6XYADVLTC75BARTLDBQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.060807082 (128h13m15.931890127s) [noise=true , address=45.91.116.8:4073 , piece_id=KNND2N5INYU5G4W75AGPOW7M7QPI6TTOZDXT3EHG3NKY5IRXZKSA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.065670312 (128h13m15.936753357s) [noise=true , address=45.91.116.13:4184 , piece_id=YGJPB7SIBL2ZQJAWUWLHHZOURQKS2HWTBETMBYLIW3A2HL4MQN3Q, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.081448576 (128h13m15.952531621s) [noise=true , address=45.91.116.14:4220 , piece_id=HLODTBHS5UASYGFQLWLG2RMDNOB6YQ2GCHXU72MUYXRHWQYWHMWQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.107956241 (128h13m15.979039286s) [noise=true , address=45.91.116.12:4240 , piece_id=PYVXH4KTJNZLDVQD4FEOCLP3EGITNB444WJNFTWOVAPJPYQV3BOQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.122140209 (128h13m15.993223254s) [noise=true , address=45.91.116.5:4321 , piece_id=BLUL5G7Z5EN6UBCXXRBICDST5FHPTEDQB6H3OZ3GD3VHAN3MIZMA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.210729766 (128h13m16.081812811s) [noise=true , address=45.91.116.9:4291 , piece_id=QT7GRV75S3K4NEEZD6XQ2DVBHVVIF7KP4LISQOOQMGXI4HCIUPRQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.916310046 (128h13m16.787393081s) [noise=true , address=145.79.227.30:4778 , piece_id=D6RSHWZZA42HTXEM672GKPC6RP7UWCCHBTEVRLA742737AR5BS4Q, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:43:13.138685713 (128h14m13.009768758s) [noise=true , address=77.90.31.12:4593 , piece_id=76TXIDKPW5M2DJQDLDNDVNP56W3ETFY53YGA6ZX575ONL6A7475Q, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: rpc: dial tcp 77.90.31.12:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139000452 (128h14m13.010083487s) [noise=true , address=77.90.31.18:4815 , piece_id=GKZVHO2OWJFF5AYFB76LFED3GLIGOOLJROLBNWINIV3KRXOGK3MA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139108908 (128h14m13.010191953s) [noise=true , address=77.90.31.16:4766 , piece_id=H74RM2MKTJHFNLEPHWTOZTK34X6LGE7NI5U4NRL3JD5T6BQAVTEQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: rpc: dial tcp 77.90.31.16:4766: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139145348 (128h14m13.010228403s) [noise=true , address=77.90.31.12:4603 , piece_id=XCPA4SH4C65HTX7AHTHDEHB2RHJFOKUWFTHYIZ35UUEO2VAYPNEQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139470748 (128h14m13.010553783s) [noise=true , address=77.90.31.13:4649 , piece_id=BNTF6775HEWBGVIHHUAVD7B73E37VWYVQTVWIK637E4GORYSC52A, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139769116 (128h14m13.010852161s) [noise=true , address=77.90.31.21:4906 , piece_id=PO3CMNXQP4XHEDE47SC276JPPVYR4TDMFFOISIQBSSKOX6GVWXIQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: write tcp 188.166.225.170:35940->77.90.31.21:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139926185 (128h14m13.01100922s) [noise=true , address=77.90.31.17:4800 , piece_id=XEXUBXCIZF4VK4Q2ALGTR5QITQOK2DDK27O4GQZF5TSZQHJVPCHQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:43:13.139951443 (128h14m13.011034478s) [seg_pos={0 0}, upload=111969]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:43:13.380665947 (128h14m13.251748982s) [upload=111969]: Done uploading segments... Dec 10 23:43:13.380708497 (128h14m13.251791542s) [upload=111969]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.456866254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [......................................XXXXX.XXXXX.X..XXXXXXX] duration: 1m0.244422138s trace: https://jaeger.dev.storj.io/trace/5336b456b3dc690c FAIL (2025-12-10 23:43:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 583014s) [seg_pos={0 0}, upload=111969]: Begin upload segment... Dec 10 23:42:14.080581605 (128h13m13.95166464s) [seg_pos={0 0}, upload=111969]: Done begin upload segment. Dec 10 23:42:14.080652671 (128h13m13.951735706s) [noise=true , address=145.79.227.30:4778 , piece_id=D6RSHWZZA42HTXEM672GKPC6RP7UWCCHBTEVRLA742737AR5BS4Q, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.080677046 (128h13m13.951760091s) [noise=true , address=45.91.116.12:4240 , piece_id=PYVXH4KTJNZLDVQD4FEOCLP3EGITNB444WJNFTWOVAPJPYQV3BOQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.080763732 (128h13m13.951846777s) [noise=true , address=145.79.227.24:4600 , piece_id=4Y7Z3IIUMKZPDMA5XXU7G4VMQJ2DR2PFDX7POV5JAIPU5NQNEHYQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.081152592 (128h13m13.952235627s) [noise=true , address=45.91.116.13:4184 , piece_id=YGJPB7SIBL2ZQJAWUWLHHZOURQKS2HWTBETMBYLIW3A2HL4MQN3Q, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.081477200 (128h13m13.952560235s) [noise=true , address=45.91.116.5:4321 , piece_id=BLUL5G7Z5EN6UBCXXRBICDST5FHPTEDQB6H3OZ3GD3VHAN3MIZMA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.081850130 (128h13m13.952933166s) [noise=true , address=45.91.116.8:4073 , piece_id=KNND2N5INYU5G4W75AGPOW7M7QPI6TTOZDXT3EHG3NKY5IRXZKSA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.082142687 (128h13m13.953225732s) [noise=true , address=77.90.31.12:4593 , piece_id=76TXIDKPW5M2DJQDLDNDVNP56W3ETFY53YGA6ZX575ONL6A7475Q, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.082667116 (128h13m13.953750151s) [noise=true , address=77.90.31.18:4815 , piece_id=GKZVHO2OWJFF5AYFB76LFED3GLIGOOLJROLBNWINIV3KRXOGK3MA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.083072919 (128h13m13.954155964s) [noise=true , address=45.91.116.9:4291 , piece_id=QT7GRV75S3K4NEEZD6XQ2DVBHVVIF7KP4LISQOOQMGXI4HCIUPRQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.083405002 (128h13m13.954488047s) [noise=true , address=145.79.227.51:4423 , piece_id=J3WR4E6L6S6KGQIQMC6DDDMQJEKSUMXWEBXZPIZ6GIHJNO76Z5KQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.083725111 (128h13m13.954808146s) [noise=true , address=77.90.31.13:4649 , piece_id=BNTF6775HEWBGVIHHUAVD7B73E37VWYVQTVWIK637E4GORYSC52A, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.084217679 (128h13m13.955300724s) [noise=true , address=77.90.31.17:4800 , piece_id=XEXUBXCIZF4VK4Q2ALGTR5QITQOK2DDK27O4GQZF5TSZQHJVPCHQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.084771624 (128h13m13.955854669s) [noise=true , address=77.90.31.16:4766 , piece_id=H74RM2MKTJHFNLEPHWTOZTK34X6LGE7NI5U4NRL3JD5T6BQAVTEQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.086573935 (128h13m13.95765697s) [noise=true , address=77.90.31.12:4603 , piece_id=XCPA4SH4C65HTX7AHTHDEHB2RHJFOKUWFTHYIZ35UUEO2VAYPNEQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.087995993 (128h13m13.959079038s) [noise=true , address=145.79.227.32:4845 , piece_id=GE5B7ZU5VLCRRU7IHY2SIJSD755JFGH6D5TSOAKNDHZSLVQXRMSA, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.089335203 (128h13m13.960418248s) [noise=true , address=145.79.227.38:4996 , piece_id=47M5V3FV44NVMRCWKQBUFN7ASJET4BA6I6XYADVLTC75BARTLDBQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.092096240 (128h13m13.963179276s) [noise=true , address=45.91.116.14:4220 , piece_id=HLODTBHS5UASYGFQLWLG2RMDNOB6YQ2GCHXU72MUYXRHWQYWHMWQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:14.103379665 (128h13m13.97446271s) [noise=true , address=77.90.31.21:4906 , piece_id=PO3CMNXQP4XHEDE47SC276JPPVYR4TDMFFOISIQBSSKOX6GVWXIQ, seg_pos={0 0}, upload=111969]: Uploading piece... Dec 10 23:42:15.555102291 (128h13m15.426185326s) [upload=111969]: Next returned nil segment Dec 10 23:42:15.555117991 (128h13m15.426201036s) [upload=111969]: Waiting for error group managing segments... Dec 10 23:42:15.955172339 (128h13m15.826255384s) [noise=true , address=145.79.227.24:4600 , piece_id=4Y7Z3IIUMKZPDMA5XXU7G4VMQJ2DR2PFDX7POV5JAIPU5NQNEHYQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:15.962248593 (128h13m15.833331628s) [noise=true , address=145.79.227.32:4845 , piece_id=GE5B7ZU5VLCRRU7IHY2SIJSD755JFGH6D5TSOAKNDHZSLVQXRMSA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:15.981117352 (128h13m15.852200398s) [noise=true , address=145.79.227.51:4423 , piece_id=J3WR4E6L6S6KGQIQMC6DDDMQJEKSUMXWEBXZPIZ6GIHJNO76Z5KQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.010575485 (128h13m15.88165853s) [noise=true , address=145.79.227.38:4996 , piece_id=47M5V3FV44NVMRCWKQBUFN7ASJET4BA6I6XYADVLTC75BARTLDBQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.060807082 (128h13m15.931890127s) [noise=true , address=45.91.116.8:4073 , piece_id=KNND2N5INYU5G4W75AGPOW7M7QPI6TTOZDXT3EHG3NKY5IRXZKSA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.065670312 (128h13m15.936753357s) [noise=true , address=45.91.116.13:4184 , piece_id=YGJPB7SIBL2ZQJAWUWLHHZOURQKS2HWTBETMBYLIW3A2HL4MQN3Q, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.081448576 (128h13m15.952531621s) [noise=true , address=45.91.116.14:4220 , piece_id=HLODTBHS5UASYGFQLWLG2RMDNOB6YQ2GCHXU72MUYXRHWQYWHMWQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.107956241 (128h13m15.979039286s) [noise=true , address=45.91.116.12:4240 , piece_id=PYVXH4KTJNZLDVQD4FEOCLP3EGITNB444WJNFTWOVAPJPYQV3BOQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.122140209 (128h13m15.993223254s) [noise=true , address=45.91.116.5:4321 , piece_id=BLUL5G7Z5EN6UBCXXRBICDST5FHPTEDQB6H3OZ3GD3VHAN3MIZMA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.210729766 (128h13m16.081812811s) [noise=true , address=45.91.116.9:4291 , piece_id=QT7GRV75S3K4NEEZD6XQ2DVBHVVIF7KP4LISQOOQMGXI4HCIUPRQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:42:16.916310046 (128h13m16.787393081s) [noise=true , address=145.79.227.30:4778 , piece_id=D6RSHWZZA42HTXEM672GKPC6RP7UWCCHBTEVRLA742737AR5BS4Q, seg_pos={0 0}, upload=111969]: Done uploading piece. err: <nil> Dec 10 23:43:13.138685713 (128h14m13.009768758s) [noise=true , address=77.90.31.12:4593 , piece_id=76TXIDKPW5M2DJQDLDNDVNP56W3ETFY53YGA6ZX575ONL6A7475Q, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: rpc: dial tcp 77.90.31.12:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139000452 (128h14m13.010083487s) [noise=true , address=77.90.31.18:4815 , piece_id=GKZVHO2OWJFF5AYFB76LFED3GLIGOOLJROLBNWINIV3KRXOGK3MA, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139108908 (128h14m13.010191953s) [noise=true , address=77.90.31.16:4766 , piece_id=H74RM2MKTJHFNLEPHWTOZTK34X6LGE7NI5U4NRL3JD5T6BQAVTEQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: rpc: dial tcp 77.90.31.16:4766: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139145348 (128h14m13.010228403s) [noise=true , address=77.90.31.12:4603 , piece_id=XCPA4SH4C65HTX7AHTHDEHB2RHJFOKUWFTHYIZ35UUEO2VAYPNEQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139470748 (128h14m13.010553783s) [noise=true , address=77.90.31.13:4649 , piece_id=BNTF6775HEWBGVIHHUAVD7B73E37VWYVQTVWIK637E4GORYSC52A, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139769116 (128h14m13.010852161s) [noise=true , address=77.90.31.21:4906 , piece_id=PO3CMNXQP4XHEDE47SC276JPPVYR4TDMFFOISIQBSSKOX6GVWXIQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: write tcp 188.166.225.170:35940->77.90.31.21:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:13.139926185 (128h14m13.01100922s) [noise=true , address=77.90.31.17:4800 , piece_id=XEXUBXCIZF4VK4Q2ALGTR5QITQOK2DDK27O4GQZF5TSZQHJVPCHQ, seg_pos={0 0}, upload=111969]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:43:13.139951443 (128h14m13.011034478s) [seg_pos={0 0}, upload=111969]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:43:13.380665947 (128h14m13.251748982s) [upload=111969]: Done uploading segments... Dec 10 23:43:13.380708497 (128h14m13.251791542s) [upload=111969]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.244443198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.785319447s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:43:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.785346248s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.880185073s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:42:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.880205292s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 4.227990962s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:42:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.228022131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......................................XXXXX.XXXXX.X..XXXXXX] duration: 1m0.24860435s trace: https://jaeger.dev.storj.io/trace/64ee37e5230ee47f FAIL (2025-12-10 23:42:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== nts... Dec 10 23:41:08.140591504 (128h12m8.011674549s) [upload=111954]: Got next segment. Inline: false Dec 10 23:41:08.140620099 (128h12m8.011703134s) [upload=111954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:41:08.140627873 (128h12m8.011710918s) [upload=111954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:41:08.390163843 (128h12m8.261246878s) [seg_pos={0 0}, upload=111954]: Begin upload segment... Dec 10 23:41:08.390223075 (128h12m8.261306121s) [seg_pos={0 0}, upload=111954]: Done begin upload segment. Dec 10 23:41:08.390256499 (128h12m8.261339534s) [noise=true , address=77.90.31.16:4766 , piece_id=V2SPASIAEHIYPWFSPFLV4UNYPG6XXCPZVXU65IBWOMFD22ZQA7AQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.390357992 (128h12m8.261441048s) [noise=true , address=5.231.73.2:4025 , piece_id=Y5JT6PRA5OP3XT2Y6FVH7QXKF67BWNP6WYQV6ZDJ4R6725SVMB3Q, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.390499542 (128h12m8.261582577s) [noise=true , address=77.90.31.17:4774 , piece_id=P5XAA4PSFGBQ4HB63M7ZJJGDDK4F4IDDFW2DF37GY6QIJPDURDRQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.390952294 (128h12m8.262035329s) [noise=true , address=5.231.73.5:4031 , piece_id=CUDOPG74OPY4ZYZACMIUD7B6B3WXGAY23SPYDJJYPVZOF5WQB35A, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.391436636 (128h12m8.262519671s) [noise=true , address=145.79.227.26:4657 , piece_id=XOFGUN4GAH43UXY3CWD5PVZ2NWTP4U7FBREV7IFNNCYEYWPLADTQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.391522861 (128h12m8.262605896s) [noise=true , address=77.90.31.19:4870 , piece_id=HOUXUE56AWBQ7RJTQX3F7ETRPPMVOYABIGBFJ2ENFF5A72GDALIQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.391812532 (128h12m8.262895567s) [noise=true , address=45.91.116.8:4056 , piece_id=ZKFHLCD2WJAGMHA42IDCBTH7Y6RADMONSKHWKGCANGSVJTQALJUQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.392079401 (128h12m8.263162436s) [noise=true , address=45.91.116.12:4243 , piece_id=PKLAOGRYE277NVX7VHWGEAKUL7CJU3WAUF32SHEGMVQ5CTTYSCVQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.392349545 (128h12m8.26343258s) [noise=true , address=77.90.31.15:4705 , piece_id=RUSP526DWQROLR3FZQXQVEKCSHX7R5Z6UA4W77ERSPZEYZJDJSTA, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.392663172 (128h12m8.263746207s) [noise=true , address=77.90.31.12:4620 , piece_id=GJ4WEV7U345N224BJZPC4KPFR4G2HD2JEHVRO54NLTMBN3XJ7GDQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.392946080 (128h12m8.264029125s) [noise=true , address=45.91.116.6:4008 , piece_id=SHNREVI6N4CBF2C7GZGEON7JQDSSTEV4BKMWV6SPHOMPSXMYXLWA, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.393205234 (128h12m8.264288269s) [noise=true , address=64.49.13.139:4470 , piece_id=SR2K2JTRXK2PRTD44IHXHTYO5SVHRBAZGZ6VLJ7NRDVLEPDBCYVQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.393584146 (128h12m8.264667181s) [noise=true , address=77.90.31.10:4539 , piece_id=4V3HZ5LKWGCUOYJJNTZZ5CYQ4LWGCBLSFXWL6B2RSR7SEMHVRKOQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.394100880 (128h12m8.265183915s) [noise=true , address=145.79.227.24:4600 , piece_id=N4XTYBZ4BY2BAURWJMSDKXS2NBZG2J3SYUMBJD4TEJ4BXU4T3C5A, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.394427692 (128h12m8.265510727s) [noise=true , address=145.79.227.30:4773 , piece_id=CLCOWUXA4RG6UCOLH67WXE27PKTGKZOT3WSKE7LM4GGWVT4DZUUA, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.394791314 (128h12m8.265874349s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZXY7LN3KYWFWLUAXKKEXNVSWUOD33YRWKVWUQSA4QVLAOEN2JDA, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.395216705 (128h12m8.26629975s) [noise=true , address=64.49.13.169:4706 , piece_id=VB3GZ76QED7GHQMSPWMTLP3XQCTCDPLM6CVCZYS7766NA2JSNM7Q, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:08.395598582 (128h12m8.266681657s) [noise=true , address=45.91.116.9:4272 , piece_id=6TIFKAMEZ45BKQMNTJDJKLCKGHBGSNHFWBFFS7RYKXRXGWGBRAPQ, seg_pos={0 0}, upload=111954]: Uploading piece... Dec 10 23:41:09.738824927 (128h12m9.609907972s) [upload=111954]: Next returned nil segment Dec 10 23:41:09.738842049 (128h12m9.609925084s) [upload=111954]: Waiting for error group managing segments... Dec 10 23:41:10.080068601 (128h12m9.951151646s) [noise=true , address=64.49.13.139:4470 , piece_id=SR2K2JTRXK2PRTD44IHXHTYO5SVHRBAZGZ6VLJ7NRDVLEPDBCYVQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.254845812 (128h12m10.125928857s) [noise=true , address=5.231.73.2:4025 , piece_id=Y5JT6PRA5OP3XT2Y6FVH7QXKF67BWNP6WYQV6ZDJ4R6725SVMB3Q, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.261020990 (128h12m10.132104035s) [noise=true , address=145.79.227.26:4657 , piece_id=XOFGUN4GAH43UXY3CWD5PVZ2NWTP4U7FBREV7IFNNCYEYWPLADTQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.265815769 (128h12m10.136898814s) [noise=true , address=145.79.227.24:4600 , piece_id=N4XTYBZ4BY2BAURWJMSDKXS2NBZG2J3SYUMBJD4TEJ4BXU4T3C5A, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.284174096 (128h12m10.155257131s) [noise=true , address=5.231.73.5:4031 , piece_id=CUDOPG74OPY4ZYZACMIUD7B6B3WXGAY23SPYDJJYPVZOF5WQB35A, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.382786087 (128h12m10.253869132s) [noise=true , address=45.91.116.8:4056 , piece_id=ZKFHLCD2WJAGMHA42IDCBTH7Y6RADMONSKHWKGCANGSVJTQALJUQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.410105657 (128h12m10.281188702s) [noise=true , address=45.91.116.6:4008 , piece_id=SHNREVI6N4CBF2C7GZGEON7JQDSSTEV4BKMWV6SPHOMPSXMYXLWA, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.459745869 (128h12m10.330828904s) [noise=true , address=45.91.116.9:4272 , piece_id=6TIFKAMEZ45BKQMNTJDJKLCKGHBGSNHFWBFFS7RYKXRXGWGBRAPQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:10.476223164 (128h12m10.347306199s) [noise=true , address=45.91.116.12:4243 , piece_id=PKLAOGRYE277NVX7VHWGEAKUL7CJU3WAUF32SHEGMVQ5CTTYSCVQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:11.248338687 (128h12m11.119421732s) [noise=true , address=145.79.227.30:4773 , piece_id=CLCOWUXA4RG6UCOLH67WXE27PKTGKZOT3WSKE7LM4GGWVT4DZUUA, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:25.212527723 (128h12m25.083610768s) [noise=true , address=64.49.13.169:4709 , piece_id=3ZXY7LN3KYWFWLUAXKKEXNVSWUOD33YRWKVWUQSA4QVLAOEN2JDA, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:41:27.860993485 (128h12m27.73207652s) [noise=true , address=64.49.13.169:4706 , piece_id=VB3GZ76QED7GHQMSPWMTLP3XQCTCDPLM6CVCZYS7766NA2JSNM7Q, seg_pos={0 0}, upload=111954]: Done uploading piece. err: <nil> Dec 10 23:42:07.448183687 (128h13m7.319266732s) [noise=true , address=77.90.31.17:4774 , piece_id=P5XAA4PSFGBQ4HB63M7ZJJGDDK4F4IDDFW2DF37GY6QIJPDURDRQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:54240->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:07.448739946 (128h13m7.319822991s) [noise=true , address=77.90.31.15:4705 , piece_id=RUSP526DWQROLR3FZQXQVEKCSHX7R5Z6UA4W77ERSPZEYZJDJSTA, seg_pos={0 0}, upload=111954]: Done uploading piece. err: ecclient: upload failed (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw, address:77.90.31.15:4705): protocol: write tcp 188.166.225.170:42626->77.90.31.15:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:07.449132354 (128h13m7.320215399s) [noise=true , address=77.90.31.12:4620 , piece_id=GJ4WEV7U345N224BJZPC4KPFR4G2HD2JEHVRO54NLTMBN3XJ7GDQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:42:07.449254486 (128h13m7.320337531s) [noise=true , address=77.90.31.10:4539 , piece_id=4V3HZ5LKWGCUOYJJNTZZ5CYQ4LWGCBLSFXWL6B2RSR7SEMHVRKOQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:42:07.449407147 (128h13m7.320490192s) [noise=true , address=77.90.31.16:4766 , piece_id=V2SPASIAEHIYPWFSPFLV4UNYPG6XXCPZVXU65IBWOMFD22ZQA7AQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:42:07.449942787 (128h13m7.321025842s) [noise=true , address=77.90.31.19:4870 , piece_id=HOUXUE56AWBQ7RJTQX3F7ETRPPMVOYABIGBFJ2ENFF5A72GDALIQ, seg_pos={0 0}, upload=111954]: Done uploading piece. err: ecclient: upload failed (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR, address:77.90.31.19:4870): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:42:07.450000968 (128h13m7.321084013s) [seg_pos={0 0}, upload=111954]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:42:07.694176660 (128h13m7.565259715s) [upload=111954]: Done uploading segments... Dec 10 23:42:07.694222859 (128h13m7.565305894s) [upload=111954]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.248624908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.776572735s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:41:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.776632769s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)