Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X................X......X.X..X.X............X.XX.XXX.X..] duration: 9.582174238s trace: https://jaeger.dev.storj.io/trace/658b819d1727f0df FAIL (2025-12-21 16:41:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:40:35.602370415 (385h11m35.47345346s) [upload=338074]: Starting upload Dec 21 16:40:35.602523266 (385h11m35.473606331s) [upload=338074]: Done starting upload Dec 21 16:40:35.602741862 (385h11m35.473824897s) [upload=338074]: Uploading segments... Dec 21 16:40:35.602789423 (385h11m35.473872458s) [upload=338074]: Got next segment. Inline: false Dec 21 16:40:35.602821543 (385h11m35.473904608s) [upload=338074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:40:35.602828808 (385h11m35.473911853s) [upload=338074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:40:35.931347739 (385h11m35.802430804s) [seg_pos={0 0}, upload=338074]: Begin upload segment... Dec 21 16:40:35.931401181 (385h11m35.802484227s) [seg_pos={0 0}, upload=338074]: Done begin upload segment. Dec 21 16:40:35.931433012 (385h11m35.802516057s) [noise=true , address=145.79.227.49:4338 , piece_id=KQNUMTLDTZJ3ZH2FZ4BPBMYW3QQUWSZDF2CZAF5ORJZNOWQYKDYQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.932365688 (385h11m35.803448723s) [noise=true , address=145.79.227.46:4603 , piece_id=CI7AOSTJAOX6FVYUNT6DJA2V4DFPN64K3ZYFXD466OBMW2IXYYDQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.932866021 (385h11m35.803949056s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933156143 (385h11m35.804239178s) [noise=true , address=145.79.227.50:4374 , piece_id=S4YFNT3WKABGGBQBTTF7YJS5CSWSS7FWIESMIKWSMCN3RIDGKE4A, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933455774 (385h11m35.804538819s) [noise=true , address=145.79.227.9:4366 , piece_id=MVAUEMN3YXIF5GHAVDPHY3PB3GZLTK36OEUZSBREKZVP5P4LPPWA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933745906 (385h11m35.804828941s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934084811 (385h11m35.805167846s) [noise=true , address=77.90.31.16:4736 , piece_id=SAEAEPEU24HW7U6EQUQ5BWFZDAWSSDKGCPAG3M5QLVLCQX7DSB7A, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934368151 (385h11m35.805451186s) [noise=true , address=145.79.227.48:4302 , piece_id=73SWSNYEJJRCCEBQACJUTZDNCACKMG426ORN33R2N5ZISG7JNNSQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934842685 (385h11m35.80592573s) [noise=true , address=145.79.227.8:4329 , piece_id=QMMCM6UWZAENIEJD4P6PCLGETJSTFEBCDH46SCXZ63ODURT4HCPQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.935187732 (385h11m35.806270767s) [noise=true , address=145.79.227.38:4978 , piece_id=DZTHNCNQYONWUSMXG3J62SXRRWOJVU3ELBLPCK73RSCWFKMXVGTA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.936485032 (385h11m35.807568067s) [noise=true , address=145.79.227.29:4942 , piece_id=MYPJQXMKYXM7JGLTQ54FV5TTOGO3RVIETYRZC65GTOCJ3X5DTIKA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.937602319 (385h11m35.808685364s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL57BZWXEMOXVYFHML2EOONIBE3NW27S6R7WK7PPY7PGEC7YTEJQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.938915911 (385h11m35.809998956s) [noise=true , address=145.79.227.35:4481 , piece_id=M3LF43STQL236TLGXA4M6GA5WDMFGV6ZSYRSMTA647QEGX3MJTXA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.940220475 (385h11m35.81130352s) [noise=true , address=145.79.227.41:4087 , piece_id=GJG2GVON3ZOUR74ZBKX77D2YGC64EJNIDC6DHXFL27FUNDL75URQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.941468551 (385h11m35.812551596s) [noise=true , address=145.79.227.37:4567 , piece_id=EZJKOB5AHUMHV5YPKNF5GF6TIJBNG35XEFITO3XZQQL2ZDT6IBMA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.942819835 (385h11m35.81390287s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:37.787105297 (385h11m37.658188332s) [upload=338074]: Next returned nil segment Dec 21 16:40:37.787126548 (385h11m37.658209603s) [upload=338074]: Waiting for error group managing segments... Dec 21 16:40:38.163941563 (385h11m38.035024598s) [noise=true , address=77.90.31.16:4736 , piece_id=SAEAEPEU24HW7U6EQUQ5BWFZDAWSSDKGCPAG3M5QLVLCQX7DSB7A, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.386386094 (385h11m38.257469129s) [noise=true , address=145.79.227.38:4978 , piece_id=DZTHNCNQYONWUSMXG3J62SXRRWOJVU3ELBLPCK73RSCWFKMXVGTA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.469904175 (385h11m38.340987221s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL57BZWXEMOXVYFHML2EOONIBE3NW27S6R7WK7PPY7PGEC7YTEJQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.510947027 (385h11m38.382030072s) [noise=true , address=145.79.227.8:4329 , piece_id=QMMCM6UWZAENIEJD4P6PCLGETJSTFEBCDH46SCXZ63ODURT4HCPQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.574352682 (385h11m38.445435727s) [noise=true , address=145.79.227.46:4603 , piece_id=CI7AOSTJAOX6FVYUNT6DJA2V4DFPN64K3ZYFXD466OBMW2IXYYDQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.672906514 (385h11m38.543989549s) [noise=true , address=145.79.227.48:4302 , piece_id=73SWSNYEJJRCCEBQACJUTZDNCACKMG426ORN33R2N5ZISG7JNNSQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.864432079 (385h11m38.735515114s) [noise=true , address=145.79.227.41:4087 , piece_id=GJG2GVON3ZOUR74ZBKX77D2YGC64EJNIDC6DHXFL27FUNDL75URQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.939644115 (385h11m38.81072715s) [noise=true , address=145.79.227.35:4481 , piece_id=M3LF43STQL236TLGXA4M6GA5WDMFGV6ZSYRSMTA647QEGX3MJTXA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.154595928 (385h11m39.025678973s) [noise=true , address=145.79.227.29:4942 , piece_id=MYPJQXMKYXM7JGLTQ54FV5TTOGO3RVIETYRZC65GTOCJ3X5DTIKA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.193955084 (385h11m39.065038119s) [noise=true , address=145.79.227.37:4567 , piece_id=EZJKOB5AHUMHV5YPKNF5GF6TIJBNG35XEFITO3XZQQL2ZDT6IBMA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.221848447 (385h11m39.092931492s) [noise=true , address=145.79.227.49:4338 , piece_id=KQNUMTLDTZJ3ZH2FZ4BPBMYW3QQUWSZDF2CZAF5ORJZNOWQYKDYQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.082093922 (385h11m39.953176977s) [noise=true , address=145.79.227.9:4366 , piece_id=MVAUEMN3YXIF5GHAVDPHY3PB3GZLTK36OEUZSBREKZVP5P4LPPWA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.122674603 (385h11m39.993757648s) [noise=true , address=145.79.227.50:4374 , piece_id=S4YFNT3WKABGGBQBTTF7YJS5CSWSS7FWIESMIKWSMCN3RIDGKE4A, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.122698218 (385h11m39.993781263s) [seg_pos={0 0}, upload=338074]: Segment reached optimal threshold of 13 pieces. Dec 21 16:40:40.122926924 (385h11m39.994009969s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection Dec 21 16:40:40.122935290 (385h11m39.994018335s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123284695 (385h11m39.99436774s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 16:40:40.123296147 (385h11m39.994379182s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123301447 (385h11m39.994384482s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 21 16:40:40.123311006 (385h11m39.994394051s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123327326 (385h11m39.994410371s) [seg_pos={0 0}, upload=338074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:40:40.123461963 (385h11m39.994544998s) [upload=338074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:40:40.123502660 (385h11m39.994585695s) [upload=338074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:40:40.484189005 (385h11m40.35527204s) [upload=338074]: Done uploading segments... Dec 21 16:40:40.484207881 (385h11m40.355290916s) [upload=338074]: Upload finished. err: <nil> Duration: 9.582203862s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X.X...........X................................] duration: 5.900939599s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:41:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.90096657s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............X.............X........X........................] duration: 2.160705266s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:41:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.160760062s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X................X......X.X..X.X............X.XX.XXX.X.] duration: 16.188181774s trace: https://jaeger.dev.storj.io/trace/658b819d1727f0df FAIL (2025-12-21 16:41:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:40:35.602370415 (385h11m35.47345346s) [upload=338074]: Starting upload Dec 21 16:40:35.602523266 (385h11m35.473606331s) [upload=338074]: Done starting upload Dec 21 16:40:35.602741862 (385h11m35.473824897s) [upload=338074]: Uploading segments... Dec 21 16:40:35.602789423 (385h11m35.473872458s) [upload=338074]: Got next segment. Inline: false Dec 21 16:40:35.602821543 (385h11m35.473904608s) [upload=338074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:40:35.602828808 (385h11m35.473911853s) [upload=338074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:40:35.931347739 (385h11m35.802430804s) [seg_pos={0 0}, upload=338074]: Begin upload segment... Dec 21 16:40:35.931401181 (385h11m35.802484227s) [seg_pos={0 0}, upload=338074]: Done begin upload segment. Dec 21 16:40:35.931433012 (385h11m35.802516057s) [noise=true , address=145.79.227.49:4338 , piece_id=KQNUMTLDTZJ3ZH2FZ4BPBMYW3QQUWSZDF2CZAF5ORJZNOWQYKDYQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.932365688 (385h11m35.803448723s) [noise=true , address=145.79.227.46:4603 , piece_id=CI7AOSTJAOX6FVYUNT6DJA2V4DFPN64K3ZYFXD466OBMW2IXYYDQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.932866021 (385h11m35.803949056s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933156143 (385h11m35.804239178s) [noise=true , address=145.79.227.50:4374 , piece_id=S4YFNT3WKABGGBQBTTF7YJS5CSWSS7FWIESMIKWSMCN3RIDGKE4A, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933455774 (385h11m35.804538819s) [noise=true , address=145.79.227.9:4366 , piece_id=MVAUEMN3YXIF5GHAVDPHY3PB3GZLTK36OEUZSBREKZVP5P4LPPWA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933745906 (385h11m35.804828941s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934084811 (385h11m35.805167846s) [noise=true , address=77.90.31.16:4736 , piece_id=SAEAEPEU24HW7U6EQUQ5BWFZDAWSSDKGCPAG3M5QLVLCQX7DSB7A, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934368151 (385h11m35.805451186s) [noise=true , address=145.79.227.48:4302 , piece_id=73SWSNYEJJRCCEBQACJUTZDNCACKMG426ORN33R2N5ZISG7JNNSQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934842685 (385h11m35.80592573s) [noise=true , address=145.79.227.8:4329 , piece_id=QMMCM6UWZAENIEJD4P6PCLGETJSTFEBCDH46SCXZ63ODURT4HCPQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.935187732 (385h11m35.806270767s) [noise=true , address=145.79.227.38:4978 , piece_id=DZTHNCNQYONWUSMXG3J62SXRRWOJVU3ELBLPCK73RSCWFKMXVGTA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.936485032 (385h11m35.807568067s) [noise=true , address=145.79.227.29:4942 , piece_id=MYPJQXMKYXM7JGLTQ54FV5TTOGO3RVIETYRZC65GTOCJ3X5DTIKA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.937602319 (385h11m35.808685364s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL57BZWXEMOXVYFHML2EOONIBE3NW27S6R7WK7PPY7PGEC7YTEJQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.938915911 (385h11m35.809998956s) [noise=true , address=145.79.227.35:4481 , piece_id=M3LF43STQL236TLGXA4M6GA5WDMFGV6ZSYRSMTA647QEGX3MJTXA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.940220475 (385h11m35.81130352s) [noise=true , address=145.79.227.41:4087 , piece_id=GJG2GVON3ZOUR74ZBKX77D2YGC64EJNIDC6DHXFL27FUNDL75URQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.941468551 (385h11m35.812551596s) [noise=true , address=145.79.227.37:4567 , piece_id=EZJKOB5AHUMHV5YPKNF5GF6TIJBNG35XEFITO3XZQQL2ZDT6IBMA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.942819835 (385h11m35.81390287s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:37.787105297 (385h11m37.658188332s) [upload=338074]: Next returned nil segment Dec 21 16:40:37.787126548 (385h11m37.658209603s) [upload=338074]: Waiting for error group managing segments... Dec 21 16:40:38.163941563 (385h11m38.035024598s) [noise=true , address=77.90.31.16:4736 , piece_id=SAEAEPEU24HW7U6EQUQ5BWFZDAWSSDKGCPAG3M5QLVLCQX7DSB7A, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.386386094 (385h11m38.257469129s) [noise=true , address=145.79.227.38:4978 , piece_id=DZTHNCNQYONWUSMXG3J62SXRRWOJVU3ELBLPCK73RSCWFKMXVGTA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.469904175 (385h11m38.340987221s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL57BZWXEMOXVYFHML2EOONIBE3NW27S6R7WK7PPY7PGEC7YTEJQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.510947027 (385h11m38.382030072s) [noise=true , address=145.79.227.8:4329 , piece_id=QMMCM6UWZAENIEJD4P6PCLGETJSTFEBCDH46SCXZ63ODURT4HCPQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.574352682 (385h11m38.445435727s) [noise=true , address=145.79.227.46:4603 , piece_id=CI7AOSTJAOX6FVYUNT6DJA2V4DFPN64K3ZYFXD466OBMW2IXYYDQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.672906514 (385h11m38.543989549s) [noise=true , address=145.79.227.48:4302 , piece_id=73SWSNYEJJRCCEBQACJUTZDNCACKMG426ORN33R2N5ZISG7JNNSQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.864432079 (385h11m38.735515114s) [noise=true , address=145.79.227.41:4087 , piece_id=GJG2GVON3ZOUR74ZBKX77D2YGC64EJNIDC6DHXFL27FUNDL75URQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.939644115 (385h11m38.81072715s) [noise=true , address=145.79.227.35:4481 , piece_id=M3LF43STQL236TLGXA4M6GA5WDMFGV6ZSYRSMTA647QEGX3MJTXA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.154595928 (385h11m39.025678973s) [noise=true , address=145.79.227.29:4942 , piece_id=MYPJQXMKYXM7JGLTQ54FV5TTOGO3RVIETYRZC65GTOCJ3X5DTIKA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.193955084 (385h11m39.065038119s) [noise=true , address=145.79.227.37:4567 , piece_id=EZJKOB5AHUMHV5YPKNF5GF6TIJBNG35XEFITO3XZQQL2ZDT6IBMA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.221848447 (385h11m39.092931492s) [noise=true , address=145.79.227.49:4338 , piece_id=KQNUMTLDTZJ3ZH2FZ4BPBMYW3QQUWSZDF2CZAF5ORJZNOWQYKDYQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.082093922 (385h11m39.953176977s) [noise=true , address=145.79.227.9:4366 , piece_id=MVAUEMN3YXIF5GHAVDPHY3PB3GZLTK36OEUZSBREKZVP5P4LPPWA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.122674603 (385h11m39.993757648s) [noise=true , address=145.79.227.50:4374 , piece_id=S4YFNT3WKABGGBQBTTF7YJS5CSWSS7FWIESMIKWSMCN3RIDGKE4A, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.122698218 (385h11m39.993781263s) [seg_pos={0 0}, upload=338074]: Segment reached optimal threshold of 13 pieces. Dec 21 16:40:40.122926924 (385h11m39.994009969s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection Dec 21 16:40:40.122935290 (385h11m39.994018335s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123284695 (385h11m39.99436774s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 16:40:40.123296147 (385h11m39.994379182s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123301447 (385h11m39.994384482s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 21 16:40:40.123311006 (385h11m39.994394051s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123327326 (385h11m39.994410371s) [seg_pos={0 0}, upload=338074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:40:40.123461963 (385h11m39.994544998s) [upload=338074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:40:40.123502660 (385h11m39.994585695s) [upload=338074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:40:40.484189005 (385h11m40.35527204s) [upload=338074]: Done uploading segments... Dec 21 16:40:40.484207881 (385h11m40.355290916s) [upload=338074]: Upload finished. err: <nil> Duration: 16.188200459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.....X................X......X.X..X.X............X.XX.XXX.X] duration: 30.004373484s trace: https://jaeger.dev.storj.io/trace/658b819d1727f0df FAIL (2025-12-21 16:41:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:40:35.602370415 (385h11m35.47345346s) [upload=338074]: Starting upload Dec 21 16:40:35.602523266 (385h11m35.473606331s) [upload=338074]: Done starting upload Dec 21 16:40:35.602741862 (385h11m35.473824897s) [upload=338074]: Uploading segments... Dec 21 16:40:35.602789423 (385h11m35.473872458s) [upload=338074]: Got next segment. Inline: false Dec 21 16:40:35.602821543 (385h11m35.473904608s) [upload=338074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:40:35.602828808 (385h11m35.473911853s) [upload=338074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:40:35.931347739 (385h11m35.802430804s) [seg_pos={0 0}, upload=338074]: Begin upload segment... Dec 21 16:40:35.931401181 (385h11m35.802484227s) [seg_pos={0 0}, upload=338074]: Done begin upload segment. Dec 21 16:40:35.931433012 (385h11m35.802516057s) [noise=true , address=145.79.227.49:4338 , piece_id=KQNUMTLDTZJ3ZH2FZ4BPBMYW3QQUWSZDF2CZAF5ORJZNOWQYKDYQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.932365688 (385h11m35.803448723s) [noise=true , address=145.79.227.46:4603 , piece_id=CI7AOSTJAOX6FVYUNT6DJA2V4DFPN64K3ZYFXD466OBMW2IXYYDQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.932866021 (385h11m35.803949056s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933156143 (385h11m35.804239178s) [noise=true , address=145.79.227.50:4374 , piece_id=S4YFNT3WKABGGBQBTTF7YJS5CSWSS7FWIESMIKWSMCN3RIDGKE4A, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933455774 (385h11m35.804538819s) [noise=true , address=145.79.227.9:4366 , piece_id=MVAUEMN3YXIF5GHAVDPHY3PB3GZLTK36OEUZSBREKZVP5P4LPPWA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.933745906 (385h11m35.804828941s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934084811 (385h11m35.805167846s) [noise=true , address=77.90.31.16:4736 , piece_id=SAEAEPEU24HW7U6EQUQ5BWFZDAWSSDKGCPAG3M5QLVLCQX7DSB7A, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934368151 (385h11m35.805451186s) [noise=true , address=145.79.227.48:4302 , piece_id=73SWSNYEJJRCCEBQACJUTZDNCACKMG426ORN33R2N5ZISG7JNNSQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.934842685 (385h11m35.80592573s) [noise=true , address=145.79.227.8:4329 , piece_id=QMMCM6UWZAENIEJD4P6PCLGETJSTFEBCDH46SCXZ63ODURT4HCPQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.935187732 (385h11m35.806270767s) [noise=true , address=145.79.227.38:4978 , piece_id=DZTHNCNQYONWUSMXG3J62SXRRWOJVU3ELBLPCK73RSCWFKMXVGTA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.936485032 (385h11m35.807568067s) [noise=true , address=145.79.227.29:4942 , piece_id=MYPJQXMKYXM7JGLTQ54FV5TTOGO3RVIETYRZC65GTOCJ3X5DTIKA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.937602319 (385h11m35.808685364s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL57BZWXEMOXVYFHML2EOONIBE3NW27S6R7WK7PPY7PGEC7YTEJQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.938915911 (385h11m35.809998956s) [noise=true , address=145.79.227.35:4481 , piece_id=M3LF43STQL236TLGXA4M6GA5WDMFGV6ZSYRSMTA647QEGX3MJTXA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.940220475 (385h11m35.81130352s) [noise=true , address=145.79.227.41:4087 , piece_id=GJG2GVON3ZOUR74ZBKX77D2YGC64EJNIDC6DHXFL27FUNDL75URQ, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.941468551 (385h11m35.812551596s) [noise=true , address=145.79.227.37:4567 , piece_id=EZJKOB5AHUMHV5YPKNF5GF6TIJBNG35XEFITO3XZQQL2ZDT6IBMA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:35.942819835 (385h11m35.81390287s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Uploading piece... Dec 21 16:40:37.787105297 (385h11m37.658188332s) [upload=338074]: Next returned nil segment Dec 21 16:40:37.787126548 (385h11m37.658209603s) [upload=338074]: Waiting for error group managing segments... Dec 21 16:40:38.163941563 (385h11m38.035024598s) [noise=true , address=77.90.31.16:4736 , piece_id=SAEAEPEU24HW7U6EQUQ5BWFZDAWSSDKGCPAG3M5QLVLCQX7DSB7A, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.386386094 (385h11m38.257469129s) [noise=true , address=145.79.227.38:4978 , piece_id=DZTHNCNQYONWUSMXG3J62SXRRWOJVU3ELBLPCK73RSCWFKMXVGTA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.469904175 (385h11m38.340987221s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL57BZWXEMOXVYFHML2EOONIBE3NW27S6R7WK7PPY7PGEC7YTEJQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.510947027 (385h11m38.382030072s) [noise=true , address=145.79.227.8:4329 , piece_id=QMMCM6UWZAENIEJD4P6PCLGETJSTFEBCDH46SCXZ63ODURT4HCPQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.574352682 (385h11m38.445435727s) [noise=true , address=145.79.227.46:4603 , piece_id=CI7AOSTJAOX6FVYUNT6DJA2V4DFPN64K3ZYFXD466OBMW2IXYYDQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.672906514 (385h11m38.543989549s) [noise=true , address=145.79.227.48:4302 , piece_id=73SWSNYEJJRCCEBQACJUTZDNCACKMG426ORN33R2N5ZISG7JNNSQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.864432079 (385h11m38.735515114s) [noise=true , address=145.79.227.41:4087 , piece_id=GJG2GVON3ZOUR74ZBKX77D2YGC64EJNIDC6DHXFL27FUNDL75URQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:38.939644115 (385h11m38.81072715s) [noise=true , address=145.79.227.35:4481 , piece_id=M3LF43STQL236TLGXA4M6GA5WDMFGV6ZSYRSMTA647QEGX3MJTXA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.154595928 (385h11m39.025678973s) [noise=true , address=145.79.227.29:4942 , piece_id=MYPJQXMKYXM7JGLTQ54FV5TTOGO3RVIETYRZC65GTOCJ3X5DTIKA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.193955084 (385h11m39.065038119s) [noise=true , address=145.79.227.37:4567 , piece_id=EZJKOB5AHUMHV5YPKNF5GF6TIJBNG35XEFITO3XZQQL2ZDT6IBMA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:39.221848447 (385h11m39.092931492s) [noise=true , address=145.79.227.49:4338 , piece_id=KQNUMTLDTZJ3ZH2FZ4BPBMYW3QQUWSZDF2CZAF5ORJZNOWQYKDYQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.082093922 (385h11m39.953176977s) [noise=true , address=145.79.227.9:4366 , piece_id=MVAUEMN3YXIF5GHAVDPHY3PB3GZLTK36OEUZSBREKZVP5P4LPPWA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.122674603 (385h11m39.993757648s) [noise=true , address=145.79.227.50:4374 , piece_id=S4YFNT3WKABGGBQBTTF7YJS5CSWSS7FWIESMIKWSMCN3RIDGKE4A, seg_pos={0 0}, upload=338074]: Done uploading piece. err: <nil> Dec 21 16:40:40.122698218 (385h11m39.993781263s) [seg_pos={0 0}, upload=338074]: Segment reached optimal threshold of 13 pieces. Dec 21 16:40:40.122926924 (385h11m39.994009969s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35038->145.79.227.36:4519: use of closed network connection Dec 21 16:40:40.122935290 (385h11m39.994018335s) [noise=true , address=145.79.227.36:4519 , piece_id=2ATUCY74OXHXPYJZLO74AYNCGTOZATLXP3GKUPLABOIQV7PY4VIQ, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123284695 (385h11m39.99436774s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 16:40:40.123296147 (385h11m39.994379182s) [noise=true , address=145.79.227.44:4194 , piece_id=SNVHXPGRGJ7TJQD27Q5B5CQGNTBWPWOADD24LGYPMDRBU7X2RYLA, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123301447 (385h11m39.994384482s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 21 16:40:40.123311006 (385h11m39.994394051s) [noise=true , address=145.79.227.40:4050 , piece_id=2PTP7GHXQSS52I5SREXB5MWHVBSGE2IMOGEZBVXAXKNEYMUUSPHA, seg_pos={0 0}, upload=338074]: Long tail cancellation due to optimal threshold Dec 21 16:40:40.123327326 (385h11m39.994410371s) [seg_pos={0 0}, upload=338074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:40:40.123461963 (385h11m39.994544998s) [upload=338074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:40:40.123502660 (385h11m39.994585695s) [upload=338074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:40:40.484189005 (385h11m40.35527204s) [upload=338074]: Done uploading segments... Dec 21 16:40:40.484207881 (385h11m40.355290916s) [upload=338074]: Upload finished. err: <nil> Duration: 30.004397459s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.X.X...........X...............................] duration: 6.097067588s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:40:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.097093838s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............X..................] duration: 732.363386ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:40:46 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 732.393463ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 748.059034ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:40:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 748.082808ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.X.X...........X..............................] duration: 5.670676677s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:40:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.670702286s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............X.................] duration: 765.264196ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:40:16 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 765.297448ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............X.............X........X.......................] duration: 2.170208713s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:40:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.170274209s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.....X................X......X.X..X.X............X.XX.XXX.] duration: 6.29836161s trace: https://jaeger.dev.storj.io/trace/24bfb0974935e421 FAIL (2025-12-21 16:40:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ZFJP6X6ULE5NKHZN3HHYVQQKVABD2XKJARIOY3UEPGOCNPS2EZMA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 36OT5PQANH3DY3KYMG3MQS3XSZT2IOPFHB5VJJIU3ONEXOE5VIGA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: YLVBB43ZVHXUJ7IA7GRCELFZI7COWGSKWNEJ36NYE5VO4ZRTMVXA) context deadline exceeded ======LOGS====== Dec 21 16:39:22.884264331 (385h10m22.755347376s) [upload=338058]: Starting upload Dec 21 16:39:22.884366896 (385h10m22.755449941s) [upload=338058]: Done starting upload Dec 21 16:39:22.884478618 (385h10m22.755561654s) [upload=338058]: Uploading segments... Dec 21 16:39:22.884695090 (385h10m22.755778136s) [upload=338058]: Got next segment. Inline: false Dec 21 16:39:22.884836269 (385h10m22.755919304s) [upload=338058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:39:22.884844335 (385h10m22.75592737s) [upload=338058]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:39:23.161357540 (385h10m23.032440575s) [seg_pos={0 0}, upload=338058]: Begin upload segment... Dec 21 16:39:23.161410772 (385h10m23.032493817s) [seg_pos={0 0}, upload=338058]: Done begin upload segment. Dec 21 16:39:23.161449455 (385h10m23.0325325s) [noise=true , address=45.91.116.11:4117 , piece_id=36OT5PQANH3DY3KYMG3MQS3XSZT2IOPFHB5VJJIU3ONEXOE5VIGA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.162346674 (385h10m23.033429709s) [noise=true , address=145.79.227.8:4329 , piece_id=CG2BWVUMZIIPHFWYIDI6ELBNS5ATZ4XBDM6G5QD7PBXVWFVC26VQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.163464853 (385h10m23.034547888s) [noise=true , address=145.79.227.12:4765 , piece_id=CCS75EKRPOJF4KK5D2ZN3YJ5YPLH7YO32RPXSF7AMCQWDIW4LO4Q, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.163832504 (385h10m23.034915539s) [noise=true , address=145.79.227.40:4049 , piece_id=S77FAAB4BJWTZMVW3TDJIVZ5IVSRD7OJ6TYG4TKLWUQCYNOGAYKQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.164181929 (385h10m23.035264964s) [noise=true , address=145.79.227.30:4762 , piece_id=PVNQBNLVHZQAZUCVAMZ7DKJ24I5KKW7EWIDMN6MLZ74Y32NOMR2A, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.164511686 (385h10m23.035594721s) [noise=true , address=45.91.116.9:4263 , piece_id=ZFJP6X6ULE5NKHZN3HHYVQQKVABD2XKJARIOY3UEPGOCNPS2EZMA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.164875530 (385h10m23.035958575s) [noise=true , address=145.79.227.48:4303 , piece_id=IBEEIY4DBO2OLWXPPST7HQOSDPS2SNQGJUBVH773ERBY7XPJXK4A, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.165931180 (385h10m23.037014215s) [noise=true , address=145.79.227.42:4123 , piece_id=FPIPZY5U4Z6KGH5IIY2EZ2RQ3732R4AMLON3DU3WTOO54Y4KNGEA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.167373155 (385h10m23.0384562s) [noise=true , address=145.79.227.38:4979 , piece_id=YR5JL2QJUV6HHD5EJB5KVNLU3UEP3OYPCHUTR2IJYGD242RNAVOQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.172176030 (385h10m23.043259076s) [noise=true , address=45.91.116.12:4228 , piece_id=YLVBB43ZVHXUJ7IA7GRCELFZI7COWGSKWNEJ36NYE5VO4ZRTMVXA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.176969939 (385h10m23.048052984s) [noise=true , address=145.79.227.51:4410 , piece_id=7PRLYJTZBTKCQ5JULOGHFAMZSYB3N5PGP2L75CFNBLEFCMP4PYKA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.178169242 (385h10m23.049252297s) [noise=true , address=145.79.227.39:4013 , piece_id=476Z23YZTWTAPXD6KP6LW2PUKQYBKVZEOMEPKIN45A6ZCSQ23PEA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.178979495 (385h10m23.05006254s) [noise=true , address=145.79.227.27:4691 , piece_id=KV5VIWBDCI4KERIQSC52MAQQ6CFGINTMYAVZNV2FTHUQQ2WV2GUQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.179991011 (385h10m23.051074046s) [noise=true , address=77.90.31.10:4520 , piece_id=UG2XKVHSSDQSUJRDHCLZ6ICLIEENISCRCZVDKQWU7OY6LYIBCTIQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.181514292 (385h10m23.052597327s) [noise=true , address=145.79.227.10:4403 , piece_id=5FAH7FQPCP47HQ3DXHIM3W5XDPOPEWGY6HKFILPCFN4TSBJ3Y6JQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.182119434 (385h10m23.053202489s) [noise=true , address=145.79.227.15:4875 , piece_id=43UNIIDKLLYBNOPLY4KLR64LQJEXPAM3OY76EJ47YSXUPGA4ONFQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:25.049229450 (385h10m24.920312495s) [upload=338058]: Next returned nil segment Dec 21 16:39:25.049244749 (385h10m24.920327794s) [upload=338058]: Waiting for error group managing segments... Dec 21 16:39:25.346334345 (385h10m25.21741738s) [noise=true , address=145.79.227.12:4765 , piece_id=CCS75EKRPOJF4KK5D2ZN3YJ5YPLH7YO32RPXSF7AMCQWDIW4LO4Q, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.390801728 (385h10m25.261884773s) [noise=true , address=145.79.227.15:4875 , piece_id=43UNIIDKLLYBNOPLY4KLR64LQJEXPAM3OY76EJ47YSXUPGA4ONFQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.401178086 (385h10m25.272261121s) [noise=true , address=145.79.227.8:4329 , piece_id=CG2BWVUMZIIPHFWYIDI6ELBNS5ATZ4XBDM6G5QD7PBXVWFVC26VQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.419182439 (385h10m25.290265474s) [noise=true , address=77.90.31.10:4520 , piece_id=UG2XKVHSSDQSUJRDHCLZ6ICLIEENISCRCZVDKQWU7OY6LYIBCTIQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.455161199 (385h10m25.326244234s) [noise=true , address=45.91.116.11:4117 , piece_id=36OT5PQANH3DY3KYMG3MQS3XSZT2IOPFHB5VJJIU3ONEXOE5VIGA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.458736256 (385h10m25.329819301s) [noise=true , address=45.91.116.9:4263 , piece_id=ZFJP6X6ULE5NKHZN3HHYVQQKVABD2XKJARIOY3UEPGOCNPS2EZMA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.677719776 (385h10m25.548802811s) [noise=true , address=145.79.227.48:4303 , piece_id=IBEEIY4DBO2OLWXPPST7HQOSDPS2SNQGJUBVH773ERBY7XPJXK4A, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.804561563 (385h10m25.675644598s) [noise=true , address=145.79.227.40:4049 , piece_id=S77FAAB4BJWTZMVW3TDJIVZ5IVSRD7OJ6TYG4TKLWUQCYNOGAYKQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.063208729 (385h10m25.934291774s) [noise=true , address=145.79.227.39:4013 , piece_id=476Z23YZTWTAPXD6KP6LW2PUKQYBKVZEOMEPKIN45A6ZCSQ23PEA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.083989468 (385h10m25.955072503s) [noise=true , address=145.79.227.27:4691 , piece_id=KV5VIWBDCI4KERIQSC52MAQQ6CFGINTMYAVZNV2FTHUQQ2WV2GUQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.123413748 (385h10m25.994496793s) [noise=true , address=145.79.227.38:4979 , piece_id=YR5JL2QJUV6HHD5EJB5KVNLU3UEP3OYPCHUTR2IJYGD242RNAVOQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.136972987 (385h10m26.008056032s) [noise=true , address=145.79.227.42:4123 , piece_id=FPIPZY5U4Z6KGH5IIY2EZ2RQ3732R4AMLON3DU3WTOO54Y4KNGEA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.194105366 (385h10m26.065188411s) [noise=true , address=45.91.116.12:4228 , piece_id=YLVBB43ZVHXUJ7IA7GRCELFZI7COWGSKWNEJ36NYE5VO4ZRTMVXA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.194125554 (385h10m26.065208589s) [seg_pos={0 0}, upload=338058]: Segment reached optimal threshold of 13 pieces. Dec 21 16:39:26.194311559 (385h10m26.065394594s) [noise=true , address=145.79.227.51:4410 , piece_id=7PRLYJTZBTKCQ5JULOGHFAMZSYB3N5PGP2L75CFNBLEFCMP4PYKA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 21 16:39:26.194317590 (385h10m26.065400625s) [noise=true , address=145.79.227.51:4410 , piece_id=7PRLYJTZBTKCQ5JULOGHFAMZSYB3N5PGP2L75CFNBLEFCMP4PYKA, seg_pos={0 0}, upload=338058]: Long tail cancellation due to optimal threshold Dec 21 16:39:26.194596011 (385h10m26.065679046s) [noise=true , address=145.79.227.30:4762 , piece_id=PVNQBNLVHZQAZUCVAMZ7DKJ24I5KKW7EWIDMN6MLZ74Y32NOMR2A, seg_pos={0 0}, upload=338058]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 21 16:39:26.194602162 (385h10m26.065685197s) [noise=true , address=145.79.227.30:4762 , piece_id=PVNQBNLVHZQAZUCVAMZ7DKJ24I5KKW7EWIDMN6MLZ74Y32NOMR2A, seg_pos={0 0}, upload=338058]: Long tail cancellation due to optimal threshold Dec 21 16:39:26.194769992 (385h10m26.065853037s) [noise=true , address=145.79.227.10:4403 , piece_id=5FAH7FQPCP47HQ3DXHIM3W5XDPOPEWGY6HKFILPCFN4TSBJ3Y6JQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: read tcp 188.166.225.170:34502->145.79.227.10:4403: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:39:26.194789869 (385h10m26.065872904s) [noise=true , address=145.79.227.10:4403 , piece_id=5FAH7FQPCP47HQ3DXHIM3W5XDPOPEWGY6HKFILPCFN4TSBJ3Y6JQ, seg_pos={0 0}, upload=338058]: Long tail cancellation due to optimal threshold Dec 21 16:39:26.194819886 (385h10m26.065902932s) [seg_pos={0 0}, upload=338058]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:39:26.194915228 (385h10m26.065998273s) [upload=338058]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:39:26.194922332 (385h10m26.066005367s) [upload=338058]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:39:26.487406458 (385h10m26.358489503s) [upload=338058]: Done uploading segments... Dec 21 16:39:26.487431225 (385h10m26.35851428s) [upload=338058]: Upload finished. err: <nil> Duration: 6.298393502s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.X.X...........X.............................] duration: 6.469203933s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:39:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.469229251s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 747.481885ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:39:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 747.51062ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.....X................X......X.X..X.X............X.XX.XXX] duration: 30.003196316s trace: https://jaeger.dev.storj.io/trace/24bfb0974935e421 FAIL (2025-12-21 16:39:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ZFJP6X6ULE5NKHZN3HHYVQQKVABD2XKJARIOY3UEPGOCNPS2EZMA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 36OT5PQANH3DY3KYMG3MQS3XSZT2IOPFHB5VJJIU3ONEXOE5VIGA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: YLVBB43ZVHXUJ7IA7GRCELFZI7COWGSKWNEJ36NYE5VO4ZRTMVXA) context deadline exceeded ======LOGS====== Dec 21 16:39:22.884264331 (385h10m22.755347376s) [upload=338058]: Starting upload Dec 21 16:39:22.884366896 (385h10m22.755449941s) [upload=338058]: Done starting upload Dec 21 16:39:22.884478618 (385h10m22.755561654s) [upload=338058]: Uploading segments... Dec 21 16:39:22.884695090 (385h10m22.755778136s) [upload=338058]: Got next segment. Inline: false Dec 21 16:39:22.884836269 (385h10m22.755919304s) [upload=338058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:39:22.884844335 (385h10m22.75592737s) [upload=338058]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:39:23.161357540 (385h10m23.032440575s) [seg_pos={0 0}, upload=338058]: Begin upload segment... Dec 21 16:39:23.161410772 (385h10m23.032493817s) [seg_pos={0 0}, upload=338058]: Done begin upload segment. Dec 21 16:39:23.161449455 (385h10m23.0325325s) [noise=true , address=45.91.116.11:4117 , piece_id=36OT5PQANH3DY3KYMG3MQS3XSZT2IOPFHB5VJJIU3ONEXOE5VIGA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.162346674 (385h10m23.033429709s) [noise=true , address=145.79.227.8:4329 , piece_id=CG2BWVUMZIIPHFWYIDI6ELBNS5ATZ4XBDM6G5QD7PBXVWFVC26VQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.163464853 (385h10m23.034547888s) [noise=true , address=145.79.227.12:4765 , piece_id=CCS75EKRPOJF4KK5D2ZN3YJ5YPLH7YO32RPXSF7AMCQWDIW4LO4Q, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.163832504 (385h10m23.034915539s) [noise=true , address=145.79.227.40:4049 , piece_id=S77FAAB4BJWTZMVW3TDJIVZ5IVSRD7OJ6TYG4TKLWUQCYNOGAYKQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.164181929 (385h10m23.035264964s) [noise=true , address=145.79.227.30:4762 , piece_id=PVNQBNLVHZQAZUCVAMZ7DKJ24I5KKW7EWIDMN6MLZ74Y32NOMR2A, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.164511686 (385h10m23.035594721s) [noise=true , address=45.91.116.9:4263 , piece_id=ZFJP6X6ULE5NKHZN3HHYVQQKVABD2XKJARIOY3UEPGOCNPS2EZMA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.164875530 (385h10m23.035958575s) [noise=true , address=145.79.227.48:4303 , piece_id=IBEEIY4DBO2OLWXPPST7HQOSDPS2SNQGJUBVH773ERBY7XPJXK4A, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.165931180 (385h10m23.037014215s) [noise=true , address=145.79.227.42:4123 , piece_id=FPIPZY5U4Z6KGH5IIY2EZ2RQ3732R4AMLON3DU3WTOO54Y4KNGEA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.167373155 (385h10m23.0384562s) [noise=true , address=145.79.227.38:4979 , piece_id=YR5JL2QJUV6HHD5EJB5KVNLU3UEP3OYPCHUTR2IJYGD242RNAVOQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.172176030 (385h10m23.043259076s) [noise=true , address=45.91.116.12:4228 , piece_id=YLVBB43ZVHXUJ7IA7GRCELFZI7COWGSKWNEJ36NYE5VO4ZRTMVXA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.176969939 (385h10m23.048052984s) [noise=true , address=145.79.227.51:4410 , piece_id=7PRLYJTZBTKCQ5JULOGHFAMZSYB3N5PGP2L75CFNBLEFCMP4PYKA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.178169242 (385h10m23.049252297s) [noise=true , address=145.79.227.39:4013 , piece_id=476Z23YZTWTAPXD6KP6LW2PUKQYBKVZEOMEPKIN45A6ZCSQ23PEA, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.178979495 (385h10m23.05006254s) [noise=true , address=145.79.227.27:4691 , piece_id=KV5VIWBDCI4KERIQSC52MAQQ6CFGINTMYAVZNV2FTHUQQ2WV2GUQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.179991011 (385h10m23.051074046s) [noise=true , address=77.90.31.10:4520 , piece_id=UG2XKVHSSDQSUJRDHCLZ6ICLIEENISCRCZVDKQWU7OY6LYIBCTIQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.181514292 (385h10m23.052597327s) [noise=true , address=145.79.227.10:4403 , piece_id=5FAH7FQPCP47HQ3DXHIM3W5XDPOPEWGY6HKFILPCFN4TSBJ3Y6JQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:23.182119434 (385h10m23.053202489s) [noise=true , address=145.79.227.15:4875 , piece_id=43UNIIDKLLYBNOPLY4KLR64LQJEXPAM3OY76EJ47YSXUPGA4ONFQ, seg_pos={0 0}, upload=338058]: Uploading piece... Dec 21 16:39:25.049229450 (385h10m24.920312495s) [upload=338058]: Next returned nil segment Dec 21 16:39:25.049244749 (385h10m24.920327794s) [upload=338058]: Waiting for error group managing segments... Dec 21 16:39:25.346334345 (385h10m25.21741738s) [noise=true , address=145.79.227.12:4765 , piece_id=CCS75EKRPOJF4KK5D2ZN3YJ5YPLH7YO32RPXSF7AMCQWDIW4LO4Q, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.390801728 (385h10m25.261884773s) [noise=true , address=145.79.227.15:4875 , piece_id=43UNIIDKLLYBNOPLY4KLR64LQJEXPAM3OY76EJ47YSXUPGA4ONFQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.401178086 (385h10m25.272261121s) [noise=true , address=145.79.227.8:4329 , piece_id=CG2BWVUMZIIPHFWYIDI6ELBNS5ATZ4XBDM6G5QD7PBXVWFVC26VQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.419182439 (385h10m25.290265474s) [noise=true , address=77.90.31.10:4520 , piece_id=UG2XKVHSSDQSUJRDHCLZ6ICLIEENISCRCZVDKQWU7OY6LYIBCTIQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.455161199 (385h10m25.326244234s) [noise=true , address=45.91.116.11:4117 , piece_id=36OT5PQANH3DY3KYMG3MQS3XSZT2IOPFHB5VJJIU3ONEXOE5VIGA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.458736256 (385h10m25.329819301s) [noise=true , address=45.91.116.9:4263 , piece_id=ZFJP6X6ULE5NKHZN3HHYVQQKVABD2XKJARIOY3UEPGOCNPS2EZMA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.677719776 (385h10m25.548802811s) [noise=true , address=145.79.227.48:4303 , piece_id=IBEEIY4DBO2OLWXPPST7HQOSDPS2SNQGJUBVH773ERBY7XPJXK4A, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:25.804561563 (385h10m25.675644598s) [noise=true , address=145.79.227.40:4049 , piece_id=S77FAAB4BJWTZMVW3TDJIVZ5IVSRD7OJ6TYG4TKLWUQCYNOGAYKQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.063208729 (385h10m25.934291774s) [noise=true , address=145.79.227.39:4013 , piece_id=476Z23YZTWTAPXD6KP6LW2PUKQYBKVZEOMEPKIN45A6ZCSQ23PEA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.083989468 (385h10m25.955072503s) [noise=true , address=145.79.227.27:4691 , piece_id=KV5VIWBDCI4KERIQSC52MAQQ6CFGINTMYAVZNV2FTHUQQ2WV2GUQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.123413748 (385h10m25.994496793s) [noise=true , address=145.79.227.38:4979 , piece_id=YR5JL2QJUV6HHD5EJB5KVNLU3UEP3OYPCHUTR2IJYGD242RNAVOQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.136972987 (385h10m26.008056032s) [noise=true , address=145.79.227.42:4123 , piece_id=FPIPZY5U4Z6KGH5IIY2EZ2RQ3732R4AMLON3DU3WTOO54Y4KNGEA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.194105366 (385h10m26.065188411s) [noise=true , address=45.91.116.12:4228 , piece_id=YLVBB43ZVHXUJ7IA7GRCELFZI7COWGSKWNEJ36NYE5VO4ZRTMVXA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: <nil> Dec 21 16:39:26.194125554 (385h10m26.065208589s) [seg_pos={0 0}, upload=338058]: Segment reached optimal threshold of 13 pieces. Dec 21 16:39:26.194311559 (385h10m26.065394594s) [noise=true , address=145.79.227.51:4410 , piece_id=7PRLYJTZBTKCQ5JULOGHFAMZSYB3N5PGP2L75CFNBLEFCMP4PYKA, seg_pos={0 0}, upload=338058]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 21 16:39:26.194317590 (385h10m26.065400625s) [noise=true , address=145.79.227.51:4410 , piece_id=7PRLYJTZBTKCQ5JULOGHFAMZSYB3N5PGP2L75CFNBLEFCMP4PYKA, seg_pos={0 0}, upload=338058]: Long tail cancellation due to optimal threshold Dec 21 16:39:26.194596011 (385h10m26.065679046s) [noise=true , address=145.79.227.30:4762 , piece_id=PVNQBNLVHZQAZUCVAMZ7DKJ24I5KKW7EWIDMN6MLZ74Y32NOMR2A, seg_pos={0 0}, upload=338058]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 21 16:39:26.194602162 (385h10m26.065685197s) [noise=true , address=145.79.227.30:4762 , piece_id=PVNQBNLVHZQAZUCVAMZ7DKJ24I5KKW7EWIDMN6MLZ74Y32NOMR2A, seg_pos={0 0}, upload=338058]: Long tail cancellation due to optimal threshold Dec 21 16:39:26.194769992 (385h10m26.065853037s) [noise=true , address=145.79.227.10:4403 , piece_id=5FAH7FQPCP47HQ3DXHIM3W5XDPOPEWGY6HKFILPCFN4TSBJ3Y6JQ, seg_pos={0 0}, upload=338058]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: read tcp 188.166.225.170:34502->145.79.227.10:4403: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:39:26.194789869 (385h10m26.065872904s) [noise=true , address=145.79.227.10:4403 , piece_id=5FAH7FQPCP47HQ3DXHIM3W5XDPOPEWGY6HKFILPCFN4TSBJ3Y6JQ, seg_pos={0 0}, upload=338058]: Long tail cancellation due to optimal threshold Dec 21 16:39:26.194819886 (385h10m26.065902932s) [seg_pos={0 0}, upload=338058]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:39:26.194915228 (385h10m26.065998273s) [upload=338058]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:39:26.194922332 (385h10m26.066005367s) [upload=338058]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:39:26.487406458 (385h10m26.358489503s) [upload=338058]: Done uploading segments... Dec 21 16:39:26.487431225 (385h10m26.35851428s) [upload=338058]: Upload finished. err: <nil> Duration: 30.003212076s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.X.X...........X............................] duration: 6.376062028s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:39:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.376086806s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.....X................X......X.X..X.X............X.XX.XX] duration: 30.002913166s trace: https://jaeger.dev.storj.io/trace/5141c57227fc44d8 FAIL (2025-12-21 16:39:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: XENJ6BTA32MTUODK3LBJJW7W3YK57X345DVSVTJFCKMVD7ZHC2IA) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: YNACP7ZKY7ZHKIOVTPTCPEY4LMSNOT4U5DJ75E4GMTU3QKZVL2RQ) context deadline exceeded; piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: JCTKAVRQBDZ4KZJEX3PAM2AUHL4YDKBW2LBVPO224OJMLSGJAYNA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: F3SYNEXHAD6DRJASX2IQBEC6ASTTFI7XXLAU7GOSOUON7LST4KMQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 5GSICZSWPDFQHFPD4WCWUFSIMFGGK6N4HG7JHB5NPUVQ45TPT2IQ) context deadline exceeded; piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: EVKWFGVAMJLRGODCO3MSIGIWYTL5WOFA2S72S33SUMHTXSOJPP7A) context deadline exceeded ======LOGS====== Dec 21 16:38:52.647431555 (385h9m52.5185146s) [upload=338050]: Starting upload Dec 21 16:38:52.647501047 (385h9m52.518584092s) [upload=338050]: Done starting upload Dec 21 16:38:52.647718542 (385h9m52.518801587s) [upload=338050]: Uploading segments... Dec 21 16:38:52.647760111 (385h9m52.518843156s) [upload=338050]: Got next segment. Inline: false Dec 21 16:38:52.647800127 (385h9m52.518883172s) [upload=338050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:38:52.647805748 (385h9m52.518888783s) [upload=338050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:38:52.896195812 (385h9m52.767278857s) [seg_pos={0 0}, upload=338050]: Begin upload segment... Dec 21 16:38:52.896246108 (385h9m52.767329143s) [seg_pos={0 0}, upload=338050]: Done begin upload segment. Dec 21 16:38:52.896300701 (385h9m52.767383736s) [noise=true , address=145.79.227.29:4942 , piece_id=F3SYNEXHAD6DRJASX2IQBEC6ASTTFI7XXLAU7GOSOUON7LST4KMQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.896314669 (385h9m52.767397714s) [noise=true , address=145.79.227.51:4410 , piece_id=YNACP7ZKY7ZHKIOVTPTCPEY4LMSNOT4U5DJ75E4GMTU3QKZVL2RQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.896813879 (385h9m52.767896914s) [noise=true , address=145.79.227.42:4123 , piece_id=5GSICZSWPDFQHFPD4WCWUFSIMFGGK6N4HG7JHB5NPUVQ45TPT2IQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.897102879 (385h9m52.768185924s) [noise=true , address=45.91.116.10:4081 , piece_id=RP4VOWBGTIUFB2Y3PXPPAOR5WSRV6AACVGN2GYKPEYOE65SQ4OTQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.897833891 (385h9m52.768916936s) [noise=true , address=145.79.227.8:4329 , piece_id=JCTKAVRQBDZ4KZJEX3PAM2AUHL4YDKBW2LBVPO224OJMLSGJAYNA, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.898636158 (385h9m52.769719193s) [noise=true , address=145.79.227.45:4231 , piece_id=F6I5LBNUEJI7UFFLIGM3LMNX4KCUDKS2JYZREX4JEGDECUHZUB7A, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.898790643 (385h9m52.769873678s) [noise=true , address=145.79.227.41:4087 , piece_id=EVKWFGVAMJLRGODCO3MSIGIWYTL5WOFA2S72S33SUMHTXSOJPP7A, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.899977683 (385h9m52.771060738s) [noise=true , address=145.79.227.15:4875 , piece_id=7V56ZSY4MJJDYEMOWDRORRLZXBTYSCX3ESOPJJDSDMRFH77LHMXQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.901390463 (385h9m52.772473498s) [noise=true , address=77.90.31.14:4661 , piece_id=UTQXVRBAUCBVSGBXUGJHVT7TUEMCMWGTKPT4PH3I5BKQRNUOBHFQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.902640313 (385h9m52.773723348s) [noise=true , address=145.79.227.46:4601 , piece_id=IR7KLJ5VPGJNWH7I3K4BCZ3SEJ7RM2ART6YOEMLYJSEIUU2FWFSQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.903840679 (385h9m52.774923714s) [noise=true , address=145.79.227.38:4979 , piece_id=5SH35SUV2VHGHHLVBQNIYE2B7ED2QSI77ZCDY3SUP6Q3B4YNGTXQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.905055202 (385h9m52.776138247s) [noise=true , address=77.90.31.10:4517 , piece_id=JYDJRBYNHSPKRHEHZRBX3GKOHRM4ZJADMPGMK5TPQMY5CYVIV3HQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.905496201 (385h9m52.776579236s) [noise=true , address=145.79.227.28:4727 , piece_id=W6ORGYMSJTU4CWC7OFTLROX6OPIQ7EQADPEZSQHBTQY7HUKJYDDQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.907675471 (385h9m52.778758516s) [noise=true , address=145.79.227.17:4972 , piece_id=R6TO7ZADAWU3P6TWIPTP2PODVYJEBYSOYG2XUEXGWE45DU367KWA, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.908365074 (385h9m52.77944811s) [noise=true , address=145.79.227.50:4375 , piece_id=XENJ6BTA32MTUODK3LBJJW7W3YK57X345DVSVTJFCKMVD7ZHC2IA, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:52.909165428 (385h9m52.780248464s) [noise=true , address=145.79.227.9:4366 , piece_id=HU3ODW6I6TKECHO2GIKSCJCQTDA56OAEAP6ZP2URH4KELPYKHFXQ, seg_pos={0 0}, upload=338050]: Uploading piece... Dec 21 16:38:54.739528041 (385h9m54.610611076s) [upload=338050]: Next returned nil segment Dec 21 16:38:54.739596451 (385h9m54.610679516s) [upload=338050]: Waiting for error group managing segments... Dec 21 16:38:55.049211497 (385h9m54.920294532s) [noise=true , address=145.79.227.50:4375 , piece_id=XENJ6BTA32MTUODK3LBJJW7W3YK57X345DVSVTJFCKMVD7ZHC2IA, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.051152442 (385h9m54.922235477s) [noise=true , address=145.79.227.41:4087 , piece_id=EVKWFGVAMJLRGODCO3MSIGIWYTL5WOFA2S72S33SUMHTXSOJPP7A, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.055049924 (385h9m54.926132969s) [noise=true , address=145.79.227.51:4410 , piece_id=YNACP7ZKY7ZHKIOVTPTCPEY4LMSNOT4U5DJ75E4GMTU3QKZVL2RQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.063273623 (385h9m54.934356668s) [noise=true , address=145.79.227.29:4942 , piece_id=F3SYNEXHAD6DRJASX2IQBEC6ASTTFI7XXLAU7GOSOUON7LST4KMQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.071993718 (385h9m54.943076753s) [noise=true , address=145.79.227.42:4123 , piece_id=5GSICZSWPDFQHFPD4WCWUFSIMFGGK6N4HG7JHB5NPUVQ45TPT2IQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.100235986 (385h9m54.971319021s) [noise=true , address=145.79.227.45:4231 , piece_id=F6I5LBNUEJI7UFFLIGM3LMNX4KCUDKS2JYZREX4JEGDECUHZUB7A, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.119807843 (385h9m54.990890888s) [noise=true , address=145.79.227.38:4979 , piece_id=5SH35SUV2VHGHHLVBQNIYE2B7ED2QSI77ZCDY3SUP6Q3B4YNGTXQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.130518588 (385h9m55.001601633s) [noise=true , address=145.79.227.28:4727 , piece_id=W6ORGYMSJTU4CWC7OFTLROX6OPIQ7EQADPEZSQHBTQY7HUKJYDDQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.142585035 (385h9m55.01366808s) [noise=true , address=145.79.227.8:4329 , piece_id=JCTKAVRQBDZ4KZJEX3PAM2AUHL4YDKBW2LBVPO224OJMLSGJAYNA, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.154553084 (385h9m55.025636129s) [noise=true , address=77.90.31.14:4661 , piece_id=UTQXVRBAUCBVSGBXUGJHVT7TUEMCMWGTKPT4PH3I5BKQRNUOBHFQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.163002884 (385h9m55.034085919s) [noise=true , address=145.79.227.15:4875 , piece_id=7V56ZSY4MJJDYEMOWDRORRLZXBTYSCX3ESOPJJDSDMRFH77LHMXQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.187257007 (385h9m55.058340052s) [noise=true , address=77.90.31.10:4517 , piece_id=JYDJRBYNHSPKRHEHZRBX3GKOHRM4ZJADMPGMK5TPQMY5CYVIV3HQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.253731626 (385h9m55.124814661s) [noise=true , address=45.91.116.10:4081 , piece_id=RP4VOWBGTIUFB2Y3PXPPAOR5WSRV6AACVGN2GYKPEYOE65SQ4OTQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: <nil> Dec 21 16:38:55.253776130 (385h9m55.124859165s) [seg_pos={0 0}, upload=338050]: Segment reached optimal threshold of 13 pieces. Dec 21 16:38:55.254592024 (385h9m55.125675069s) [noise=true , address=145.79.227.46:4601 , piece_id=IR7KLJ5VPGJNWH7I3K4BCZ3SEJ7RM2ART6YOEMLYJSEIUU2FWFSQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 21 16:38:55.254602454 (385h9m55.125685499s) [noise=true , address=145.79.227.46:4601 , piece_id=IR7KLJ5VPGJNWH7I3K4BCZ3SEJ7RM2ART6YOEMLYJSEIUU2FWFSQ, seg_pos={0 0}, upload=338050]: Long tail cancellation due to optimal threshold Dec 21 16:38:55.254879632 (385h9m55.125962697s) [noise=true , address=145.79.227.17:4972 , piece_id=R6TO7ZADAWU3P6TWIPTP2PODVYJEBYSOYG2XUEXGWE45DU367KWA, seg_pos={0 0}, upload=338050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 21 16:38:55.254890642 (385h9m55.125973687s) [noise=true , address=145.79.227.17:4972 , piece_id=R6TO7ZADAWU3P6TWIPTP2PODVYJEBYSOYG2XUEXGWE45DU367KWA, seg_pos={0 0}, upload=338050]: Long tail cancellation due to optimal threshold Dec 21 16:38:55.254981686 (385h9m55.126064721s) [noise=true , address=145.79.227.9:4366 , piece_id=HU3ODW6I6TKECHO2GIKSCJCQTDA56OAEAP6ZP2URH4KELPYKHFXQ, seg_pos={0 0}, upload=338050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: expected piece hash; context canceled; EOF Dec 21 16:38:55.254991675 (385h9m55.12607473s) [noise=true , address=145.79.227.9:4366 , piece_id=HU3ODW6I6TKECHO2GIKSCJCQTDA56OAEAP6ZP2URH4KELPYKHFXQ, seg_pos={0 0}, upload=338050]: Long tail cancellation due to optimal threshold Dec 21 16:38:55.255095282 (385h9m55.126178317s) [seg_pos={0 0}, upload=338050]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:38:55.255199961 (385h9m55.126283006s) [upload=338050]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:38:55.255208217 (385h9m55.126291252s) [upload=338050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:38:55.586482636 (385h9m55.457565681s) [upload=338050]: Done uploading segments... Dec 21 16:38:55.586516090 (385h9m55.457599125s) [upload=338050]: Upload finished. err: <nil> Duration: 30.002939486s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............X................] duration: 720.045731ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:39:10 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 720.074016ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............X.............X........X......................] duration: 2.310648724s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:39:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.310688761s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 728.160605ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:38:48 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 728.187695ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.X.X...........X...........................] duration: 6.662770364s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:38:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.662799238s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 749.348588ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:38:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 749.374768ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............X.............X........X.....................] duration: 2.124646114s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:38:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.12469183s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.............X...............] duration: 748.438175ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:38:13 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 748.454787ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.....X................X......X.X..X.X............X.XX.X] duration: 30.003445636s trace: https://jaeger.dev.storj.io/trace/4ec1e05eeb3efcd4 FAIL (2025-12-21 16:38:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VXODQTV7FHS5L4TYPYY3CJBY2RWXQEMMPGXJUUOMH3CZEAM57QXQ) context deadline exceeded; piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: AZTG4U4JXDNLZDPZUV3YRF3GMO3MD4HTITFZHKCFICURMLTKRGVA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: ETZLIES25MYZWKDAZWD4AABACFX3SPZOEUHMAKHEHZCS5J4I5I5Q) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: WGP66J2MNIQYUNNN2ARQEAZQOCC2XX34PDQMR5HEK74OAZ74QC5A) context deadline exceeded; piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: 4JIXD37ZO26SLZGNORNISGD3CNBNBL4JTM6APZSCGFO63FKVG5PQ) context deadline exceeded ======LOGS====== Dec 21 16:37:40.589088753 (385h8m40.460171788s) [upload=338034]: Starting upload Dec 21 16:37:40.589143908 (385h8m40.460226953s) [upload=338034]: Done starting upload Dec 21 16:37:40.589236774 (385h8m40.460319819s) [upload=338034]: Uploading segments... Dec 21 16:37:40.589374356 (385h8m40.460457401s) [upload=338034]: Got next segment. Inline: false Dec 21 16:37:40.589410866 (385h8m40.460493911s) [upload=338034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:37:40.589415385 (385h8m40.46049843s) [upload=338034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:37:41.181337457 (385h8m41.052420502s) [seg_pos={0 0}, upload=338034]: Begin upload segment... Dec 21 16:37:41.181388864 (385h8m41.05247191s) [seg_pos={0 0}, upload=338034]: Done begin upload segment. Dec 21 16:37:41.181426576 (385h8m41.052509611s) [noise=true , address=145.79.227.10:4401 , piece_id=PKJ6D2HCWPUY2JARPSVPHAWL2RHLXS7XLLZXJJUDOEY6KVU2PITA, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.181479457 (385h8m41.052562502s) [noise=true , address=145.79.227.11:4732 , piece_id=BZBTBTHK6WRFQDCFLXHADJR4PP6EC27TJYHMUF3E3PNGAELTJYZQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.182571376 (385h8m41.053654421s) [noise=true , address=77.90.31.11:4555 , piece_id=X6ATMOGAIIWO3D4FYCQUMQOWI26I2NHT6PYIK72FWLZX2MQI34LQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.183048445 (385h8m41.05413148s) [noise=true , address=145.79.227.16:4909 , piece_id=TRW4RPQW6W2SKIXHZGX3LCFF7QC4ZWSCDBLSOBBRPPVWKZYFPLPA, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.183385286 (385h8m41.054468321s) [noise=true , address=145.79.227.27:4691 , piece_id=WGP66J2MNIQYUNNN2ARQEAZQOCC2XX34PDQMR5HEK74OAZ74QC5A, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.183717669 (385h8m41.054800714s) [noise=true , address=145.79.227.38:4977 , piece_id=6GPUU6MN6CR7BBSTMKYUTXKJLVHL7D2XAUNDQ4VIKIG4BOJWRBNA, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.184008072 (385h8m41.055091107s) [noise=true , address=145.79.227.37:4565 , piece_id=OUX4EYBGH7IFKCYHR6742743LVWWBYVZAIC5VX6JP24ECLNPHSGQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.184436328 (385h8m41.055519373s) [noise=true , address=145.79.227.7:4295 , piece_id=62C4XFSKGAKQ2ZT34YWKV22HNG4AZBHKO72DBSQPCHRPR4POLFVQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.184577827 (385h8m41.055660872s) [noise=true , address=145.79.227.32:4836 , piece_id=4JIXD37ZO26SLZGNORNISGD3CNBNBL4JTM6APZSCGFO63FKVG5PQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.185032383 (385h8m41.056115418s) [noise=true , address=145.79.227.43:4157 , piece_id=VXODQTV7FHS5L4TYPYY3CJBY2RWXQEMMPGXJUUOMH3CZEAM57QXQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.185385385 (385h8m41.05646842s) [noise=true , address=145.79.227.28:4727 , piece_id=5VWF7QGNR7UDULPRUBMBXOAWLC6FDKXHR4M2HQ65AW42K6CIAWPQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.185714210 (385h8m41.056797245s) [noise=true , address=77.90.31.13:4629 , piece_id=F5RICJUJ7KDTEBSSDERMOZQE2NVXXSS6AHWLIJJAXIPFE6XAMD5Q, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.186163927 (385h8m41.057246962s) [noise=true , address=145.79.227.8:4329 , piece_id=AZTG4U4JXDNLZDPZUV3YRF3GMO3MD4HTITFZHKCFICURMLTKRGVA, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.186229352 (385h8m41.057312397s) [noise=true , address=145.79.227.36:4517 , piece_id=ETZLIES25MYZWKDAZWD4AABACFX3SPZOEUHMAKHEHZCS5J4I5I5Q, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.186557587 (385h8m41.057640622s) [noise=true , address=145.79.227.46:4601 , piece_id=YP74XJW5DYOIZ2SFGDHTRSNPO3H36QZDDO67Y277PYLHARGBABXQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:41.186890671 (385h8m41.057973716s) [noise=true , address=145.79.227.39:4014 , piece_id=HT7VU3QWWHDUE3GF7LRYZ3R3HXLUB35642K7GXQEP34QU7N557FQ, seg_pos={0 0}, upload=338034]: Uploading piece... Dec 21 16:37:43.026934890 (385h8m42.898017925s) [upload=338034]: Next returned nil segment Dec 21 16:37:43.026948766 (385h8m42.898031801s) [upload=338034]: Waiting for error group managing segments... Dec 21 16:37:43.337327767 (385h8m43.208410802s) [noise=true , address=145.79.227.8:4329 , piece_id=AZTG4U4JXDNLZDPZUV3YRF3GMO3MD4HTITFZHKCFICURMLTKRGVA, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.337599584 (385h8m43.208682619s) [noise=true , address=145.79.227.32:4836 , piece_id=4JIXD37ZO26SLZGNORNISGD3CNBNBL4JTM6APZSCGFO63FKVG5PQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.338889180 (385h8m43.209972215s) [noise=true , address=145.79.227.16:4909 , piece_id=TRW4RPQW6W2SKIXHZGX3LCFF7QC4ZWSCDBLSOBBRPPVWKZYFPLPA, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.361042213 (385h8m43.232125248s) [noise=true , address=77.90.31.13:4629 , piece_id=F5RICJUJ7KDTEBSSDERMOZQE2NVXXSS6AHWLIJJAXIPFE6XAMD5Q, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.363633357 (385h8m43.234716402s) [noise=true , address=145.79.227.11:4732 , piece_id=BZBTBTHK6WRFQDCFLXHADJR4PP6EC27TJYHMUF3E3PNGAELTJYZQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.405115986 (385h8m43.276199021s) [noise=true , address=145.79.227.27:4691 , piece_id=WGP66J2MNIQYUNNN2ARQEAZQOCC2XX34PDQMR5HEK74OAZ74QC5A, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.416722568 (385h8m43.287805603s) [noise=true , address=145.79.227.7:4295 , piece_id=62C4XFSKGAKQ2ZT34YWKV22HNG4AZBHKO72DBSQPCHRPR4POLFVQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.428062931 (385h8m43.299145966s) [noise=true , address=145.79.227.43:4157 , piece_id=VXODQTV7FHS5L4TYPYY3CJBY2RWXQEMMPGXJUUOMH3CZEAM57QXQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.441999138 (385h8m43.313082183s) [noise=true , address=145.79.227.46:4601 , piece_id=YP74XJW5DYOIZ2SFGDHTRSNPO3H36QZDDO67Y277PYLHARGBABXQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.448327478 (385h8m43.319410523s) [noise=true , address=77.90.31.11:4555 , piece_id=X6ATMOGAIIWO3D4FYCQUMQOWI26I2NHT6PYIK72FWLZX2MQI34LQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.464251159 (385h8m43.335334194s) [noise=true , address=145.79.227.38:4977 , piece_id=6GPUU6MN6CR7BBSTMKYUTXKJLVHL7D2XAUNDQ4VIKIG4BOJWRBNA, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:43.494796212 (385h8m43.365879247s) [noise=true , address=145.79.227.39:4014 , piece_id=HT7VU3QWWHDUE3GF7LRYZ3R3HXLUB35642K7GXQEP34QU7N557FQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:44.100699147 (385h8m43.971782192s) [noise=true , address=145.79.227.36:4517 , piece_id=ETZLIES25MYZWKDAZWD4AABACFX3SPZOEUHMAKHEHZCS5J4I5I5Q, seg_pos={0 0}, upload=338034]: Done uploading piece. err: <nil> Dec 21 16:37:44.100722953 (385h8m43.971805998s) [seg_pos={0 0}, upload=338034]: Segment reached optimal threshold of 13 pieces. Dec 21 16:37:44.101085322 (385h8m43.972168367s) [noise=true , address=145.79.227.10:4401 , piece_id=PKJ6D2HCWPUY2JARPSVPHAWL2RHLXS7XLLZXJJUDOEY6KVU2PITA, seg_pos={0 0}, upload=338034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 16:37:44.101095602 (385h8m43.972178638s) [noise=true , address=145.79.227.10:4401 , piece_id=PKJ6D2HCWPUY2JARPSVPHAWL2RHLXS7XLLZXJJUDOEY6KVU2PITA, seg_pos={0 0}, upload=338034]: Long tail cancellation due to optimal threshold Dec 21 16:37:44.101474173 (385h8m43.972557209s) [noise=true , address=145.79.227.37:4565 , piece_id=OUX4EYBGH7IFKCYHR6742743LVWWBYVZAIC5VX6JP24ECLNPHSGQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 21 16:37:44.101485024 (385h8m43.972568069s) [noise=true , address=145.79.227.37:4565 , piece_id=OUX4EYBGH7IFKCYHR6742743LVWWBYVZAIC5VX6JP24ECLNPHSGQ, seg_pos={0 0}, upload=338034]: Long tail cancellation due to optimal threshold Dec 21 16:37:44.101512015 (385h8m43.972595061s) [noise=true , address=145.79.227.28:4727 , piece_id=5VWF7QGNR7UDULPRUBMBXOAWLC6FDKXHR4M2HQ65AW42K6CIAWPQ, seg_pos={0 0}, upload=338034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:56392->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:56392->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56392->145.79.227.28:4727: use of closed network connection Dec 21 16:37:44.101523567 (385h8m43.972606612s) [noise=true , address=145.79.227.28:4727 , piece_id=5VWF7QGNR7UDULPRUBMBXOAWLC6FDKXHR4M2HQ65AW42K6CIAWPQ, seg_pos={0 0}, upload=338034]: Long tail cancellation due to optimal threshold Dec 21 16:37:44.101547794 (385h8m43.972630829s) [seg_pos={0 0}, upload=338034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:37:44.101633246 (385h8m43.972716281s) [upload=338034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:37:44.101640399 (385h8m43.972723434s) [upload=338034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:37:44.427256404 (385h8m44.298339439s) [upload=338034]: Done uploading segments... Dec 21 16:37:44.427286081 (385h8m44.298369116s) [upload=338034]: Upload finished. err: <nil> Duration: 30.003465634s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.X.X...........X..........................] duration: 6.301820769s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:38:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.301872256s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.X.X...........X.........................] duration: 5.964841521s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:37:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.964880665s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [............................X...X...........................] duration: 774.241769ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:37:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 774.268811ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.............X..............] duration: 710.671812ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:37:14 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 710.70667ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X.............X........X....................] duration: 2.307671385s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:37:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.307744203s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.X.X...........X........................] duration: 6.478193321s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:37:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.478209833s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X....X.....X................X......X.X..X.X............X.XX.] duration: 7.395975857s trace: https://jaeger.dev.storj.io/trace/1016b478798cc13d FAIL (2025-12-21 16:36:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: QHQOUXJ7H2ZWUGJ2ZJGXGWDRJVU76LIUF2EFNYMZNTH2MH3P55FQ) context deadline exceeded; piecestore: (Node ID: 1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, Piece ID: HQI6NUO6CX2LDMNIHMJ2PAUBD3QHTNJI4QAXM3KU5PXTZKEACPWQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: TO6JAL5OQRPTKD3RISLM7AC7HOUIAW3NB57POB7N2EMNYEW7NAHQ) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: BENUA7VIC766PUH4MNOPC3SALSFTFLIVCTLXLMFBJX35ZEYRXE6A) context deadline exceeded; piecestore: (Node ID: 1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, Piece ID: 7ZTJD35I27NKYLCQTV4ZQS5RNJ6IQ2W3BL4WTWAHWWPTV5EKB5JQ) context deadline exceeded; piecestore: (Node ID: 129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, Piece ID: IIQPFSFBFBHSCRA5BPX3RHPVJ3UTJSRKHWIEMM3PDG4O4K6PNJKA) context deadline exceeded ======LOGS====== Dec 21 16:35:41.157045688 (385h6m41.028128733s) [upload=338005]: Starting upload Dec 21 16:35:41.157119920 (385h6m41.028202955s) [upload=338005]: Done starting upload Dec 21 16:35:41.157196336 (385h6m41.028279371s) [upload=338005]: Uploading segments... Dec 21 16:35:41.157433908 (385h6m41.028516963s) [upload=338005]: Got next segment. Inline: false Dec 21 16:35:41.157486137 (385h6m41.028569182s) [upload=338005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:35:41.157492850 (385h6m41.028575895s) [upload=338005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:35:41.466101340 (385h6m41.337184375s) [seg_pos={0 0}, upload=338005]: Begin upload segment... Dec 21 16:35:41.466163719 (385h6m41.337246754s) [seg_pos={0 0}, upload=338005]: Done begin upload segment. Dec 21 16:35:41.466193857 (385h6m41.337276902s) [noise=true , address=77.90.31.10:4517 , piece_id=FHWCIUBQ64WL7KRICL5NNFY75YA5LGTCYDLBFQUDBWE3SNOEQWDA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.466205649 (385h6m41.337288694s) [noise=true , address=145.79.227.37:4566 , piece_id=7ZTJD35I27NKYLCQTV4ZQS5RNJ6IQ2W3BL4WTWAHWWPTV5EKB5JQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.466910151 (385h6m41.337993186s) [noise=true , address=145.79.227.45:4231 , piece_id=IIQPFSFBFBHSCRA5BPX3RHPVJ3UTJSRKHWIEMM3PDG4O4K6PNJKA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.468012640 (385h6m41.339095685s) [noise=true , address=77.90.31.11:4556 , piece_id=VDFA4CJA5HWIE7N4ICRS7IB5P6N4N3RVBQLMCHRT5YYIZ4WAKMBQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.468427470 (385h6m41.339510505s) [noise=true , address=145.79.227.46:4601 , piece_id=QHQOUXJ7H2ZWUGJ2ZJGXGWDRJVU76LIUF2EFNYMZNTH2MH3P55FQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.468806441 (385h6m41.339889476s) [noise=true , address=145.79.227.27:4691 , piece_id=QPVCAAFVPGWNOCJVS4OVRQTLQ3AM4LJF3ZAY6AFYP5PAIVJSS2RQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.469286235 (385h6m41.34036928s) [noise=true , address=145.79.227.41:4086 , piece_id=HQI6NUO6CX2LDMNIHMJ2PAUBD3QHTNJI4QAXM3KU5PXTZKEACPWQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.470624353 (385h6m41.341707388s) [noise=true , address=145.79.227.51:4411 , piece_id=3JKA6EJMQIVFPMRD44HQ3MZS5JZZCGN7LWAOAGEQG2LQ3OLI7R3Q, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.470728722 (385h6m41.341811767s) [noise=true , address=145.79.227.40:4051 , piece_id=Q5UXO5U2OG5RYAPE2OIZDWIICJIQFHMDXWP7O2O6PTIG4LILE26Q, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.471897518 (385h6m41.342980553s) [noise=true , address=145.79.227.19:4078 , piece_id=M3HOJBMN3666JFZCELQNXJUKZXJFGVPMCFE7XF5KTTY5EWJIUISA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.472111886 (385h6m41.343194921s) [noise=true , address=77.90.31.14:4662 , piece_id=CN5AQ6MSYQQT3UV2XJPUXT44B4Z6CMGVATDMHKPZD3QJSCKETAYA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.473432831 (385h6m41.344515866s) [noise=true , address=145.79.227.7:4295 , piece_id=HAU4N6Z2ZC5YGIUXP5ESAPSHXLTWZJPO2MBJ35FDE56OZFBJWNMA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.473507052 (385h6m41.344590087s) [noise=true , address=145.79.227.49:4337 , piece_id=TO6JAL5OQRPTKD3RISLM7AC7HOUIAW3NB57POB7N2EMNYEW7NAHQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.474662001 (385h6m41.345745046s) [noise=true , address=145.79.227.17:4972 , piece_id=2BO6R7W6BQCO5Q2KWIVSNC2AJERYXL3MY2B4KNPT7WYIQHXNUDKA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.479950351 (385h6m41.351033386s) [noise=true , address=45.91.116.4:4939 , piece_id=YCEE5EODELW5DBMAU2S7C2JKPTBDGMKFZ5LEND2YLW7XHQNYVC2A, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.482094343 (385h6m41.353177388s) [noise=true , address=145.79.227.16:4909 , piece_id=BENUA7VIC766PUH4MNOPC3SALSFTFLIVCTLXLMFBJX35ZEYRXE6A, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:43.179781219 (385h6m43.050864254s) [upload=338005]: Next returned nil segment Dec 21 16:35:43.179795305 (385h6m43.05087834s) [upload=338005]: Waiting for error group managing segments... Dec 21 16:35:43.464741589 (385h6m43.335824624s) [noise=true , address=145.79.227.41:4086 , piece_id=HQI6NUO6CX2LDMNIHMJ2PAUBD3QHTNJI4QAXM3KU5PXTZKEACPWQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.465184422 (385h6m43.336267467s) [noise=true , address=145.79.227.49:4337 , piece_id=TO6JAL5OQRPTKD3RISLM7AC7HOUIAW3NB57POB7N2EMNYEW7NAHQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.466873910 (385h6m43.337956945s) [noise=true , address=145.79.227.19:4078 , piece_id=M3HOJBMN3666JFZCELQNXJUKZXJFGVPMCFE7XF5KTTY5EWJIUISA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.467327754 (385h6m43.338410789s) [noise=true , address=145.79.227.37:4566 , piece_id=7ZTJD35I27NKYLCQTV4ZQS5RNJ6IQ2W3BL4WTWAHWWPTV5EKB5JQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.470719662 (385h6m43.341802697s) [noise=true , address=145.79.227.45:4231 , piece_id=IIQPFSFBFBHSCRA5BPX3RHPVJ3UTJSRKHWIEMM3PDG4O4K6PNJKA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.472186646 (385h6m43.343269691s) [noise=true , address=145.79.227.7:4295 , piece_id=HAU4N6Z2ZC5YGIUXP5ESAPSHXLTWZJPO2MBJ35FDE56OZFBJWNMA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.477147633 (385h6m43.348230698s) [noise=true , address=145.79.227.27:4691 , piece_id=QPVCAAFVPGWNOCJVS4OVRQTLQ3AM4LJF3ZAY6AFYP5PAIVJSS2RQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.483403033 (385h6m43.354486068s) [noise=true , address=145.79.227.16:4909 , piece_id=BENUA7VIC766PUH4MNOPC3SALSFTFLIVCTLXLMFBJX35ZEYRXE6A, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.485496911 (385h6m43.356579946s) [noise=true , address=145.79.227.46:4601 , piece_id=QHQOUXJ7H2ZWUGJ2ZJGXGWDRJVU76LIUF2EFNYMZNTH2MH3P55FQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.512443651 (385h6m43.383526686s) [noise=true , address=145.79.227.51:4411 , piece_id=3JKA6EJMQIVFPMRD44HQ3MZS5JZZCGN7LWAOAGEQG2LQ3OLI7R3Q, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.529604960 (385h6m43.400687995s) [noise=true , address=145.79.227.17:4972 , piece_id=2BO6R7W6BQCO5Q2KWIVSNC2AJERYXL3MY2B4KNPT7WYIQHXNUDKA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.696835584 (385h6m43.567918619s) [noise=true , address=77.90.31.14:4662 , piece_id=CN5AQ6MSYQQT3UV2XJPUXT44B4Z6CMGVATDMHKPZD3QJSCKETAYA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.754724884 (385h6m43.625807919s) [noise=true , address=45.91.116.4:4939 , piece_id=YCEE5EODELW5DBMAU2S7C2JKPTBDGMKFZ5LEND2YLW7XHQNYVC2A, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.754743238 (385h6m43.625826273s) [seg_pos={0 0}, upload=338005]: Segment reached optimal threshold of 13 pieces. Dec 21 16:35:43.754977134 (385h6m43.626060169s) [noise=true , address=77.90.31.10:4517 , piece_id=FHWCIUBQ64WL7KRICL5NNFY75YA5LGTCYDLBFQUDBWE3SNOEQWDA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 21 16:35:43.754983947 (385h6m43.626066982s) [noise=true , address=77.90.31.10:4517 , piece_id=FHWCIUBQ64WL7KRICL5NNFY75YA5LGTCYDLBFQUDBWE3SNOEQWDA, seg_pos={0 0}, upload=338005]: Long tail cancellation due to optimal threshold Dec 21 16:35:43.755116840 (385h6m43.626199885s) [noise=true , address=145.79.227.40:4051 , piece_id=Q5UXO5U2OG5RYAPE2OIZDWIICJIQFHMDXWP7O2O6PTIG4LILE26Q, seg_pos={0 0}, upload=338005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 16:35:43.755129655 (385h6m43.62621269s) [noise=true , address=145.79.227.40:4051 , piece_id=Q5UXO5U2OG5RYAPE2OIZDWIICJIQFHMDXWP7O2O6PTIG4LILE26Q, seg_pos={0 0}, upload=338005]: Long tail cancellation due to optimal threshold Dec 21 16:35:43.755241568 (385h6m43.626324613s) [noise=true , address=77.90.31.11:4556 , piece_id=VDFA4CJA5HWIE7N4ICRS7IB5P6N4N3RVBQLMCHRT5YYIZ4WAKMBQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:37856->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:37856->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37856->77.90.31.11:4556: use of closed network connection Dec 21 16:35:43.755248300 (385h6m43.626331335s) [noise=true , address=77.90.31.11:4556 , piece_id=VDFA4CJA5HWIE7N4ICRS7IB5P6N4N3RVBQLMCHRT5YYIZ4WAKMBQ, seg_pos={0 0}, upload=338005]: Long tail cancellation due to optimal threshold Dec 21 16:35:43.755264240 (385h6m43.626347275s) [seg_pos={0 0}, upload=338005]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:35:43.755325898 (385h6m43.626408933s) [upload=338005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:35:43.755330396 (385h6m43.626413431s) [upload=338005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:35:44.046604529 (385h6m43.917687574s) [upload=338005]: Done uploading segments... Dec 21 16:35:44.046624226 (385h6m43.917707271s) [upload=338005]: Upload finished. err: <nil> Duration: 7.396006425s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.X.X...........X.......................] duration: 6.138055312s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:36:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.138080211s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration: 729.244045ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:36:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 729.27281ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [................................X.............X.............] duration: 759.057416ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:36:13 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 759.097143ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X.............X........X...................] duration: 2.261872386s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:36:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.261923594s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX....X.....X................X......X.X..X.X............X.XX] duration: 30.003617969s trace: https://jaeger.dev.storj.io/trace/1016b478798cc13d FAIL (2025-12-21 16:36:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: QHQOUXJ7H2ZWUGJ2ZJGXGWDRJVU76LIUF2EFNYMZNTH2MH3P55FQ) context deadline exceeded; piecestore: (Node ID: 1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, Piece ID: HQI6NUO6CX2LDMNIHMJ2PAUBD3QHTNJI4QAXM3KU5PXTZKEACPWQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: TO6JAL5OQRPTKD3RISLM7AC7HOUIAW3NB57POB7N2EMNYEW7NAHQ) context deadline exceeded; piecestore: (Node ID: 12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, Piece ID: BENUA7VIC766PUH4MNOPC3SALSFTFLIVCTLXLMFBJX35ZEYRXE6A) context deadline exceeded; piecestore: (Node ID: 1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, Piece ID: 7ZTJD35I27NKYLCQTV4ZQS5RNJ6IQ2W3BL4WTWAHWWPTV5EKB5JQ) context deadline exceeded; piecestore: (Node ID: 129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, Piece ID: IIQPFSFBFBHSCRA5BPX3RHPVJ3UTJSRKHWIEMM3PDG4O4K6PNJKA) context deadline exceeded ======LOGS====== Dec 21 16:35:41.157045688 (385h6m41.028128733s) [upload=338005]: Starting upload Dec 21 16:35:41.157119920 (385h6m41.028202955s) [upload=338005]: Done starting upload Dec 21 16:35:41.157196336 (385h6m41.028279371s) [upload=338005]: Uploading segments... Dec 21 16:35:41.157433908 (385h6m41.028516963s) [upload=338005]: Got next segment. Inline: false Dec 21 16:35:41.157486137 (385h6m41.028569182s) [upload=338005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:35:41.157492850 (385h6m41.028575895s) [upload=338005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:35:41.466101340 (385h6m41.337184375s) [seg_pos={0 0}, upload=338005]: Begin upload segment... Dec 21 16:35:41.466163719 (385h6m41.337246754s) [seg_pos={0 0}, upload=338005]: Done begin upload segment. Dec 21 16:35:41.466193857 (385h6m41.337276902s) [noise=true , address=77.90.31.10:4517 , piece_id=FHWCIUBQ64WL7KRICL5NNFY75YA5LGTCYDLBFQUDBWE3SNOEQWDA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.466205649 (385h6m41.337288694s) [noise=true , address=145.79.227.37:4566 , piece_id=7ZTJD35I27NKYLCQTV4ZQS5RNJ6IQ2W3BL4WTWAHWWPTV5EKB5JQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.466910151 (385h6m41.337993186s) [noise=true , address=145.79.227.45:4231 , piece_id=IIQPFSFBFBHSCRA5BPX3RHPVJ3UTJSRKHWIEMM3PDG4O4K6PNJKA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.468012640 (385h6m41.339095685s) [noise=true , address=77.90.31.11:4556 , piece_id=VDFA4CJA5HWIE7N4ICRS7IB5P6N4N3RVBQLMCHRT5YYIZ4WAKMBQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.468427470 (385h6m41.339510505s) [noise=true , address=145.79.227.46:4601 , piece_id=QHQOUXJ7H2ZWUGJ2ZJGXGWDRJVU76LIUF2EFNYMZNTH2MH3P55FQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.468806441 (385h6m41.339889476s) [noise=true , address=145.79.227.27:4691 , piece_id=QPVCAAFVPGWNOCJVS4OVRQTLQ3AM4LJF3ZAY6AFYP5PAIVJSS2RQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.469286235 (385h6m41.34036928s) [noise=true , address=145.79.227.41:4086 , piece_id=HQI6NUO6CX2LDMNIHMJ2PAUBD3QHTNJI4QAXM3KU5PXTZKEACPWQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.470624353 (385h6m41.341707388s) [noise=true , address=145.79.227.51:4411 , piece_id=3JKA6EJMQIVFPMRD44HQ3MZS5JZZCGN7LWAOAGEQG2LQ3OLI7R3Q, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.470728722 (385h6m41.341811767s) [noise=true , address=145.79.227.40:4051 , piece_id=Q5UXO5U2OG5RYAPE2OIZDWIICJIQFHMDXWP7O2O6PTIG4LILE26Q, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.471897518 (385h6m41.342980553s) [noise=true , address=145.79.227.19:4078 , piece_id=M3HOJBMN3666JFZCELQNXJUKZXJFGVPMCFE7XF5KTTY5EWJIUISA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.472111886 (385h6m41.343194921s) [noise=true , address=77.90.31.14:4662 , piece_id=CN5AQ6MSYQQT3UV2XJPUXT44B4Z6CMGVATDMHKPZD3QJSCKETAYA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.473432831 (385h6m41.344515866s) [noise=true , address=145.79.227.7:4295 , piece_id=HAU4N6Z2ZC5YGIUXP5ESAPSHXLTWZJPO2MBJ35FDE56OZFBJWNMA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.473507052 (385h6m41.344590087s) [noise=true , address=145.79.227.49:4337 , piece_id=TO6JAL5OQRPTKD3RISLM7AC7HOUIAW3NB57POB7N2EMNYEW7NAHQ, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.474662001 (385h6m41.345745046s) [noise=true , address=145.79.227.17:4972 , piece_id=2BO6R7W6BQCO5Q2KWIVSNC2AJERYXL3MY2B4KNPT7WYIQHXNUDKA, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.479950351 (385h6m41.351033386s) [noise=true , address=45.91.116.4:4939 , piece_id=YCEE5EODELW5DBMAU2S7C2JKPTBDGMKFZ5LEND2YLW7XHQNYVC2A, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:41.482094343 (385h6m41.353177388s) [noise=true , address=145.79.227.16:4909 , piece_id=BENUA7VIC766PUH4MNOPC3SALSFTFLIVCTLXLMFBJX35ZEYRXE6A, seg_pos={0 0}, upload=338005]: Uploading piece... Dec 21 16:35:43.179781219 (385h6m43.050864254s) [upload=338005]: Next returned nil segment Dec 21 16:35:43.179795305 (385h6m43.05087834s) [upload=338005]: Waiting for error group managing segments... Dec 21 16:35:43.464741589 (385h6m43.335824624s) [noise=true , address=145.79.227.41:4086 , piece_id=HQI6NUO6CX2LDMNIHMJ2PAUBD3QHTNJI4QAXM3KU5PXTZKEACPWQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.465184422 (385h6m43.336267467s) [noise=true , address=145.79.227.49:4337 , piece_id=TO6JAL5OQRPTKD3RISLM7AC7HOUIAW3NB57POB7N2EMNYEW7NAHQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.466873910 (385h6m43.337956945s) [noise=true , address=145.79.227.19:4078 , piece_id=M3HOJBMN3666JFZCELQNXJUKZXJFGVPMCFE7XF5KTTY5EWJIUISA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.467327754 (385h6m43.338410789s) [noise=true , address=145.79.227.37:4566 , piece_id=7ZTJD35I27NKYLCQTV4ZQS5RNJ6IQ2W3BL4WTWAHWWPTV5EKB5JQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.470719662 (385h6m43.341802697s) [noise=true , address=145.79.227.45:4231 , piece_id=IIQPFSFBFBHSCRA5BPX3RHPVJ3UTJSRKHWIEMM3PDG4O4K6PNJKA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.472186646 (385h6m43.343269691s) [noise=true , address=145.79.227.7:4295 , piece_id=HAU4N6Z2ZC5YGIUXP5ESAPSHXLTWZJPO2MBJ35FDE56OZFBJWNMA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.477147633 (385h6m43.348230698s) [noise=true , address=145.79.227.27:4691 , piece_id=QPVCAAFVPGWNOCJVS4OVRQTLQ3AM4LJF3ZAY6AFYP5PAIVJSS2RQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.483403033 (385h6m43.354486068s) [noise=true , address=145.79.227.16:4909 , piece_id=BENUA7VIC766PUH4MNOPC3SALSFTFLIVCTLXLMFBJX35ZEYRXE6A, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.485496911 (385h6m43.356579946s) [noise=true , address=145.79.227.46:4601 , piece_id=QHQOUXJ7H2ZWUGJ2ZJGXGWDRJVU76LIUF2EFNYMZNTH2MH3P55FQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.512443651 (385h6m43.383526686s) [noise=true , address=145.79.227.51:4411 , piece_id=3JKA6EJMQIVFPMRD44HQ3MZS5JZZCGN7LWAOAGEQG2LQ3OLI7R3Q, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.529604960 (385h6m43.400687995s) [noise=true , address=145.79.227.17:4972 , piece_id=2BO6R7W6BQCO5Q2KWIVSNC2AJERYXL3MY2B4KNPT7WYIQHXNUDKA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.696835584 (385h6m43.567918619s) [noise=true , address=77.90.31.14:4662 , piece_id=CN5AQ6MSYQQT3UV2XJPUXT44B4Z6CMGVATDMHKPZD3QJSCKETAYA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.754724884 (385h6m43.625807919s) [noise=true , address=45.91.116.4:4939 , piece_id=YCEE5EODELW5DBMAU2S7C2JKPTBDGMKFZ5LEND2YLW7XHQNYVC2A, seg_pos={0 0}, upload=338005]: Done uploading piece. err: <nil> Dec 21 16:35:43.754743238 (385h6m43.625826273s) [seg_pos={0 0}, upload=338005]: Segment reached optimal threshold of 13 pieces. Dec 21 16:35:43.754977134 (385h6m43.626060169s) [noise=true , address=77.90.31.10:4517 , piece_id=FHWCIUBQ64WL7KRICL5NNFY75YA5LGTCYDLBFQUDBWE3SNOEQWDA, seg_pos={0 0}, upload=338005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 21 16:35:43.754983947 (385h6m43.626066982s) [noise=true , address=77.90.31.10:4517 , piece_id=FHWCIUBQ64WL7KRICL5NNFY75YA5LGTCYDLBFQUDBWE3SNOEQWDA, seg_pos={0 0}, upload=338005]: Long tail cancellation due to optimal threshold Dec 21 16:35:43.755116840 (385h6m43.626199885s) [noise=true , address=145.79.227.40:4051 , piece_id=Q5UXO5U2OG5RYAPE2OIZDWIICJIQFHMDXWP7O2O6PTIG4LILE26Q, seg_pos={0 0}, upload=338005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 16:35:43.755129655 (385h6m43.62621269s) [noise=true , address=145.79.227.40:4051 , piece_id=Q5UXO5U2OG5RYAPE2OIZDWIICJIQFHMDXWP7O2O6PTIG4LILE26Q, seg_pos={0 0}, upload=338005]: Long tail cancellation due to optimal threshold Dec 21 16:35:43.755241568 (385h6m43.626324613s) [noise=true , address=77.90.31.11:4556 , piece_id=VDFA4CJA5HWIE7N4ICRS7IB5P6N4N3RVBQLMCHRT5YYIZ4WAKMBQ, seg_pos={0 0}, upload=338005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:37856->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:37856->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37856->77.90.31.11:4556: use of closed network connection Dec 21 16:35:43.755248300 (385h6m43.626331335s) [noise=true , address=77.90.31.11:4556 , piece_id=VDFA4CJA5HWIE7N4ICRS7IB5P6N4N3RVBQLMCHRT5YYIZ4WAKMBQ, seg_pos={0 0}, upload=338005]: Long tail cancellation due to optimal threshold Dec 21 16:35:43.755264240 (385h6m43.626347275s) [seg_pos={0 0}, upload=338005]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:35:43.755325898 (385h6m43.626408933s) [upload=338005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:35:43.755330396 (385h6m43.626413431s) [upload=338005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:35:44.046604529 (385h6m43.917687574s) [upload=338005]: Done uploading segments... Dec 21 16:35:44.046624226 (385h6m43.917707271s) [upload=338005]: Upload finished. err: <nil> Duration: 30.00363999s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.X.X...........X......................] duration: 6.33783737s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:36:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.337866415s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration: 754.145493ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:36:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 754.171662ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X.............X........X..................] duration: 2.270836283s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:35:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.270960628s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.X.X...........X.....................] duration: 6.217650141s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:35:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.217672725s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.............X............] duration: 737.723194ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:35:18 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 737.750274ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XX....X.....X................X......X.X..X.X............X.X] duration: 30.003072034s trace: https://jaeger.dev.storj.io/trace/530bb0666593d32d FAIL (2025-12-21 16:35:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: EKWCWTARLCYERQT6IJK4V7LRPKBJY3WZLW6IBI5YY76IVJPJOHUA) context deadline exceeded; piecestore: (Node ID: 12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, Piece ID: DVX2SD2JTKF6UQBTHOBVKTXZDUS27TO4MYWDBKILXF2ML63M5ROQ) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: QFVG2H33ZKHTSFTOY3ZXSSRLEXSCZPZLDOGVYL2HO5GU4FSOIHIQ) context deadline exceeded ======LOGS====== Dec 21 16:34:44.820207173 (385h5m44.691290218s) [upload=337993]: Starting upload Dec 21 16:34:44.820293097 (385h5m44.691376142s) [upload=337993]: Done starting upload Dec 21 16:34:44.820406673 (385h5m44.691489708s) [upload=337993]: Uploading segments... Dec 21 16:34:44.820551949 (385h5m44.691634984s) [upload=337993]: Got next segment. Inline: false Dec 21 16:34:44.820603398 (385h5m44.691686443s) [upload=337993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:34:44.820612084 (385h5m44.691695119s) [upload=337993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:34:45.328256069 (385h5m45.199339114s) [seg_pos={0 0}, upload=337993]: Begin upload segment... Dec 21 16:34:45.328301275 (385h5m45.19938431s) [seg_pos={0 0}, upload=337993]: Done begin upload segment. Dec 21 16:34:45.328330801 (385h5m45.199413836s) [noise=true , address=45.91.116.13:4156 , piece_id=J7O3T2KRAXYCWA45XFFOTTHDDK3HQUUZFALGELWQ2UU7GAJYBVZA, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.328425351 (385h5m45.199508416s) [noise=true , address=145.79.227.44:4193 , piece_id=C5UXRWKSGKLINTY4DVZFQ7Q73KYP5V5F6ZVVNMQFLXNQNJJFGSNQ, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.328950671 (385h5m45.200033707s) [noise=true , address=145.79.227.35:4483 , piece_id=DAKKH6SUTBSR72WITZEKSRS4TX5UZEKEZFP4D4KUQLFN3QEUMLKQ, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.329250261 (385h5m45.200333306s) [noise=true , address=145.79.227.29:4941 , piece_id=U4BQF6R3LRYJ5LN3PMKD3D3PDYEGXEBTUCPWGHMQV3KR657NHH4Q, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.329521097 (385h5m45.200604132s) [noise=true , address=77.90.31.21:4880 , piece_id=77BGWVRWL3GL2TDYPABWPF5EG435AHZG6EHPYGADI6FGSL5EBR4A, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.329784269 (385h5m45.200867304s) [noise=true , address=145.79.227.48:4303 , piece_id=DVX2SD2JTKF6UQBTHOBVKTXZDUS27TO4MYWDBKILXF2ML63M5ROQ, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.330131690 (385h5m45.201214725s) [noise=true , address=145.79.227.10:4403 , piece_id=G7TD2RXKUYQAUSKNZX3BFTC3ADUUZZBCWKRKCIO777K2FKWVARSA, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.330453062 (385h5m45.201536097s) [noise=true , address=45.91.116.14:4190 , piece_id=J3I5Z36HNMWVAPVJONVEVCBBXEHFA224B5RNHI3QUUE2QGZBMD7Q, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.330495453 (385h5m45.201578488s) [noise=true , address=77.90.31.14:4664 , piece_id=C5LYFUGOPIFQ5ZNUB7N67PSOHTERK53FFYFH5HN7WI5QZZK7KN7Q, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.330755007 (385h5m45.201838042s) [noise=true , address=145.79.227.32:4835 , piece_id=QFVG2H33ZKHTSFTOY3ZXSSRLEXSCZPZLDOGVYL2HO5GU4FSOIHIQ, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.330917065 (385h5m45.2020001s) [noise=true , address=145.79.227.47:4266 , piece_id=HGDRSZMHMA67LH6UOUJEFV6YFATGUFVKLVL5TANNWA776SVGLTBQ, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.331243568 (385h5m45.202326603s) [noise=true , address=145.79.227.52:4445 , piece_id=LPGVBQ2H7DZ2LR6MC3TQGABPG65VVMC25DTT2LYHLFZ5WTC5FS4Q, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.331569378 (385h5m45.202652423s) [noise=true , address=45.91.116.5:4299 , piece_id=64DHJ2P6LXTGO6GKTL6RXBY7M6YOZSKG73IK7FNGRINFWNBFTIZA, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.331881232 (385h5m45.202964267s) [noise=true , address=77.90.31.12:4590 , piece_id=S5JEF75IJHU4YL55RSFUUJ5XKPFA22FWUEZBMMQSKDZXY2I65EAA, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.332174189 (385h5m45.203257234s) [noise=true , address=145.79.227.50:4373 , piece_id=EKWCWTARLCYERQT6IJK4V7LRPKBJY3WZLW6IBI5YY76IVJPJOHUA, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:45.332253340 (385h5m45.203336375s) [noise=true , address=145.79.227.49:4337 , piece_id=AG3GFND2TREBZZ52CZFPESUVKLFIBPHTZRF24SN42677KUYAYRXQ, seg_pos={0 0}, upload=337993]: Uploading piece... Dec 21 16:34:47.168481782 (385h5m47.039564817s) [upload=337993]: Next returned nil segment Dec 21 16:34:47.168495888 (385h5m47.039578933s) [upload=337993]: Waiting for error group managing segments... Dec 21 16:34:47.521788815 (385h5m47.39287185s) [noise=true , address=145.79.227.49:4337 , piece_id=AG3GFND2TREBZZ52CZFPESUVKLFIBPHTZRF24SN42677KUYAYRXQ, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.532216831 (385h5m47.403299866s) [noise=true , address=77.90.31.14:4664 , piece_id=C5LYFUGOPIFQ5ZNUB7N67PSOHTERK53FFYFH5HN7WI5QZZK7KN7Q, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.551072937 (385h5m47.422155992s) [noise=true , address=145.79.227.52:4445 , piece_id=LPGVBQ2H7DZ2LR6MC3TQGABPG65VVMC25DTT2LYHLFZ5WTC5FS4Q, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.562980240 (385h5m47.434063285s) [noise=true , address=45.91.116.14:4190 , piece_id=J3I5Z36HNMWVAPVJONVEVCBBXEHFA224B5RNHI3QUUE2QGZBMD7Q, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.566113475 (385h5m47.43719651s) [noise=true , address=77.90.31.12:4590 , piece_id=S5JEF75IJHU4YL55RSFUUJ5XKPFA22FWUEZBMMQSKDZXY2I65EAA, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.569035569 (385h5m47.440118615s) [noise=true , address=145.79.227.50:4373 , piece_id=EKWCWTARLCYERQT6IJK4V7LRPKBJY3WZLW6IBI5YY76IVJPJOHUA, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.573704921 (385h5m47.444787956s) [noise=true , address=77.90.31.21:4880 , piece_id=77BGWVRWL3GL2TDYPABWPF5EG435AHZG6EHPYGADI6FGSL5EBR4A, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.583147572 (385h5m47.454230607s) [noise=true , address=145.79.227.47:4266 , piece_id=HGDRSZMHMA67LH6UOUJEFV6YFATGUFVKLVL5TANNWA776SVGLTBQ, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.598500034 (385h5m47.46958308s) [noise=true , address=45.91.116.5:4299 , piece_id=64DHJ2P6LXTGO6GKTL6RXBY7M6YOZSKG73IK7FNGRINFWNBFTIZA, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.923904066 (385h5m47.794987101s) [noise=true , address=145.79.227.32:4835 , piece_id=QFVG2H33ZKHTSFTOY3ZXSSRLEXSCZPZLDOGVYL2HO5GU4FSOIHIQ, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:47.959964831 (385h5m47.831047876s) [noise=true , address=145.79.227.48:4303 , piece_id=DVX2SD2JTKF6UQBTHOBVKTXZDUS27TO4MYWDBKILXF2ML63M5ROQ, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:48.187086197 (385h5m48.058169242s) [noise=true , address=145.79.227.10:4403 , piece_id=G7TD2RXKUYQAUSKNZX3BFTC3ADUUZZBCWKRKCIO777K2FKWVARSA, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:48.291720256 (385h5m48.162803312s) [noise=true , address=145.79.227.35:4483 , piece_id=DAKKH6SUTBSR72WITZEKSRS4TX5UZEKEZFP4D4KUQLFN3QEUMLKQ, seg_pos={0 0}, upload=337993]: Done uploading piece. err: <nil> Dec 21 16:34:48.291740165 (385h5m48.16282321s) [seg_pos={0 0}, upload=337993]: Segment reached optimal threshold of 13 pieces. Dec 21 16:34:48.291919256 (385h5m48.163002291s) [noise=true , address=45.91.116.13:4156 , piece_id=J7O3T2KRAXYCWA45XFFOTTHDDK3HQUUZFALGELWQ2UU7GAJYBVZA, seg_pos={0 0}, upload=337993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 21 16:34:48.291926569 (385h5m48.163009615s) [noise=true , address=45.91.116.13:4156 , piece_id=J7O3T2KRAXYCWA45XFFOTTHDDK3HQUUZFALGELWQ2UU7GAJYBVZA, seg_pos={0 0}, upload=337993]: Long tail cancellation due to optimal threshold Dec 21 16:34:48.292080572 (385h5m48.163163607s) [noise=true , address=145.79.227.44:4193 , piece_id=C5UXRWKSGKLINTY4DVZFQ7Q73KYP5V5F6ZVVNMQFLXNQNJJFGSNQ, seg_pos={0 0}, upload=337993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 21 16:34:48.292088458 (385h5m48.163171493s) [noise=true , address=145.79.227.44:4193 , piece_id=C5UXRWKSGKLINTY4DVZFQ7Q73KYP5V5F6ZVVNMQFLXNQNJJFGSNQ, seg_pos={0 0}, upload=337993]: Long tail cancellation due to optimal threshold Dec 21 16:34:48.292111030 (385h5m48.163194085s) [noise=true , address=145.79.227.29:4941 , piece_id=U4BQF6R3LRYJ5LN3PMKD3D3PDYEGXEBTUCPWGHMQV3KR657NHH4Q, seg_pos={0 0}, upload=337993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 21 16:34:48.292118965 (385h5m48.16320202s) [noise=true , address=145.79.227.29:4941 , piece_id=U4BQF6R3LRYJ5LN3PMKD3D3PDYEGXEBTUCPWGHMQV3KR657NHH4Q, seg_pos={0 0}, upload=337993]: Long tail cancellation due to optimal threshold Dec 21 16:34:48.292150135 (385h5m48.16323318s) [seg_pos={0 0}, upload=337993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:34:48.292279481 (385h5m48.163362536s) [upload=337993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:34:48.292286615 (385h5m48.16336966s) [upload=337993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:34:48.608234249 (385h5m48.479317294s) [upload=337993]: Done uploading segments... Dec 21 16:34:48.608248307 (385h5m48.479331342s) [upload=337993]: Upload finished. err: <nil> Duration: 30.003092784s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...X........................] duration: 740.545256ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:35:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 740.563541ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.............X...........] duration: 723.877788ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:34:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 723.897857ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.X.X...........X....................] duration: 6.010811162s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:34:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.010858792s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................X.............X........X.................] duration: 2.371850171s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:34:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.371877002s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 710.968249ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:34:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 710.996351ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.............X..........] duration: 741.37554ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:34:01 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 741.405146ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XX....X.....X................X......X.X..X.X............X.] duration: 9.145747568s trace: https://jaeger.dev.storj.io/trace/527133954b47f654 FAIL (2025-12-21 16:33:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 16:33:14.013946247 (385h4m13.885029282s) [upload=337969]: Starting upload Dec 21 16:33:14.014004078 (385h4m13.885087113s) [upload=337969]: Done starting upload Dec 21 16:33:14.014097206 (385h4m13.885180251s) [upload=337969]: Uploading segments... Dec 21 16:33:14.014246780 (385h4m13.885329825s) [upload=337969]: Got next segment. Inline: false Dec 21 16:33:14.014308909 (385h4m13.885391954s) [upload=337969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:33:14.014316914 (385h4m13.885399969s) [upload=337969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:33:14.593979323 (385h4m14.465062358s) [seg_pos={0 0}, upload=337969]: Begin upload segment... Dec 21 16:33:14.594023647 (385h4m14.465106682s) [seg_pos={0 0}, upload=337969]: Done begin upload segment. Dec 21 16:33:14.594052542 (385h4m14.465135587s) [noise=true , address=145.79.227.45:4231 , piece_id=5GC5TU6SIZ6PWWWNLPTU747EGCVFALDGAFV56QQL4GC2WWQTT6AQ, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.594164906 (385h4m14.465247941s) [noise=true , address=145.79.227.52:4445 , piece_id=MPBM5MUIE6JMOH5NVJZQ2EO55IQ2U2EGXDFPGXG74WZPFKHPQKAA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.594772183 (385h4m14.465855218s) [noise=true , address=145.79.227.36:4518 , piece_id=7ZSV7H2ZAFWOMGSMDQRHNJBLOZUZRQDABXIU5APB2SPZC5GLWAUA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.595370783 (385h4m14.466453828s) [noise=true , address=145.79.227.35:4483 , piece_id=JYMPMBCN7Z5XCT36TJH74SHKBU52DSPTJDCZGR4LE5PZOG34RW3A, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.595983088 (385h4m14.467066123s) [noise=true , address=145.79.227.38:4978 , piece_id=MLRTOXRTPJX55KIMLBW3YL4PGB3TAIFQNYALNCRY4WSGN4Q5ZL6Q, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.596215821 (385h4m14.467298867s) [noise=true , address=145.79.227.44:4195 , piece_id=B63MEZQU2FS5QIH4JXGCD6Y7QZNP2C3DNN7XIGDRDNCOJLERQMSA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.596967824 (385h4m14.468050869s) [noise=true , address=45.91.116.9:4263 , piece_id=XTTRW2C3ZAFEHM25XD2UDVKAFZQKUU22NZBAFZ75FAWERKITYRXA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.597087030 (385h4m14.468170075s) [noise=true , address=145.79.227.8:4329 , piece_id=PF5OR7AK52ZXC36N4WUHP43BJWO56MD42JOH6IY67HO3GJ6CHBEQ, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.597609916 (385h4m14.468692951s) [noise=true , address=145.79.227.11:4731 , piece_id=YQBNGW6RP2ONNNDCA7LXA5GGZSJ6WSALH7IAIVTOPOM7K4TKAGOQ, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.597956716 (385h4m14.469039752s) [noise=true , address=145.79.227.32:4835 , piece_id=YHZYZJKJ76CVMLKJMWY3LVPHLSWKOCIVXSJ7SSAJYDS5DUQKMHZQ, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.598445838 (385h4m14.469528883s) [noise=true , address=145.79.227.37:4566 , piece_id=LBYIB2OTYS4JVQMK6LKMJNSO3BVVD7WBZRPOQ7S2T2TQYDV2DNVA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.598642112 (385h4m14.469725147s) [noise=true , address=145.79.227.42:4123 , piece_id=CMAKIFMFMVSTF2YSILUT426J363CT3DJYWBWFMCKF6Y6KL5TC2AA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.598808348 (385h4m14.469891383s) [noise=true , address=145.79.227.27:4690 , piece_id=A62SD74ONGGIE4YRVJGFZQMACGXVIID2ECVDQ56AOVC4CF2C5ZNQ, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.599113369 (385h4m14.470196404s) [noise=true , address=145.79.227.34:4906 , piece_id=H4NB7LRW5QGNJ2OP6RQELYC2DZOCMGVO3IBGU7VHJAKZU5DBMJDA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.599280978 (385h4m14.470364023s) [noise=true , address=45.91.116.7:4011 , piece_id=EB45LIL2UV2QRC4UZMJRZMCEQFN6NZLIRIXHOC2QEQSTUFDNKWAQ, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:14.599904886 (385h4m14.470987931s) [noise=true , address=45.91.116.10:4084 , piece_id=YFMJE75NDR5F775HHMSQHANZ4Y7ZPYT5RY5GIBKUKEQUO2NPQEIA, seg_pos={0 0}, upload=337969]: Uploading piece... Dec 21 16:33:16.465023056 (385h4m16.336106091s) [upload=337969]: Next returned nil segment Dec 21 16:33:16.465060347 (385h4m16.336143382s) [upload=337969]: Waiting for error group managing segments... Dec 21 16:33:16.808083518 (385h4m16.679166563s) [noise=true , address=145.79.227.8:4329 , piece_id=PF5OR7AK52ZXC36N4WUHP43BJWO56MD42JOH6IY67HO3GJ6CHBEQ, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:16.825183569 (385h4m16.696266614s) [noise=true , address=145.79.227.42:4123 , piece_id=CMAKIFMFMVSTF2YSILUT426J363CT3DJYWBWFMCKF6Y6KL5TC2AA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:16.826367784 (385h4m16.697450819s) [noise=true , address=145.79.227.35:4483 , piece_id=JYMPMBCN7Z5XCT36TJH74SHKBU52DSPTJDCZGR4LE5PZOG34RW3A, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:16.838434922 (385h4m16.709517957s) [noise=true , address=145.79.227.27:4690 , piece_id=A62SD74ONGGIE4YRVJGFZQMACGXVIID2ECVDQ56AOVC4CF2C5ZNQ, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:16.868273549 (385h4m16.739356594s) [noise=true , address=145.79.227.45:4231 , piece_id=5GC5TU6SIZ6PWWWNLPTU747EGCVFALDGAFV56QQL4GC2WWQTT6AQ, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:16.899273478 (385h4m16.770356513s) [noise=true , address=45.91.116.10:4084 , piece_id=YFMJE75NDR5F775HHMSQHANZ4Y7ZPYT5RY5GIBKUKEQUO2NPQEIA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:16.911994761 (385h4m16.783077796s) [noise=true , address=45.91.116.7:4011 , piece_id=EB45LIL2UV2QRC4UZMJRZMCEQFN6NZLIRIXHOC2QEQSTUFDNKWAQ, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:16.912600234 (385h4m16.783683269s) [noise=true , address=45.91.116.9:4263 , piece_id=XTTRW2C3ZAFEHM25XD2UDVKAFZQKUU22NZBAFZ75FAWERKITYRXA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:17.150363233 (385h4m17.021446278s) [noise=true , address=145.79.227.52:4445 , piece_id=MPBM5MUIE6JMOH5NVJZQ2EO55IQ2U2EGXDFPGXG74WZPFKHPQKAA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:17.490942669 (385h4m17.362025704s) [noise=true , address=145.79.227.11:4731 , piece_id=YQBNGW6RP2ONNNDCA7LXA5GGZSJ6WSALH7IAIVTOPOM7K4TKAGOQ, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:17.919802937 (385h4m17.790885972s) [noise=true , address=145.79.227.37:4566 , piece_id=LBYIB2OTYS4JVQMK6LKMJNSO3BVVD7WBZRPOQ7S2T2TQYDV2DNVA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:17.920824533 (385h4m17.791907578s) [noise=true , address=145.79.227.34:4906 , piece_id=H4NB7LRW5QGNJ2OP6RQELYC2DZOCMGVO3IBGU7VHJAKZU5DBMJDA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:18.060666063 (385h4m17.931749109s) [noise=true , address=145.79.227.32:4835 , piece_id=YHZYZJKJ76CVMLKJMWY3LVPHLSWKOCIVXSJ7SSAJYDS5DUQKMHZQ, seg_pos={0 0}, upload=337969]: Done uploading piece. err: <nil> Dec 21 16:33:18.060686442 (385h4m17.931769487s) [seg_pos={0 0}, upload=337969]: Segment reached optimal threshold of 13 pieces. Dec 21 16:33:18.060900310 (385h4m17.931983345s) [noise=true , address=145.79.227.44:4195 , piece_id=B63MEZQU2FS5QIH4JXGCD6Y7QZNP2C3DNN7XIGDRDNCOJLERQMSA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 16:33:18.060910649 (385h4m17.931993694s) [noise=true , address=145.79.227.44:4195 , piece_id=B63MEZQU2FS5QIH4JXGCD6Y7QZNP2C3DNN7XIGDRDNCOJLERQMSA, seg_pos={0 0}, upload=337969]: Long tail cancellation due to optimal threshold Dec 21 16:33:18.060953030 (385h4m17.932036075s) [noise=true , address=145.79.227.36:4518 , piece_id=7ZSV7H2ZAFWOMGSMDQRHNJBLOZUZRQDABXIU5APB2SPZC5GLWAUA, seg_pos={0 0}, upload=337969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: write tcp 188.166.225.170:53444->145.79.227.36:4518: use of closed network connection; write tcp 188.166.225.170:53444->145.79.227.36:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->145.79.227.36:4518: use of closed network connection Dec 21 16:33:18.060963349 (385h4m17.932046384s) [noise=true , address=145.79.227.36:4518 , piece_id=7ZSV7H2ZAFWOMGSMDQRHNJBLOZUZRQDABXIU5APB2SPZC5GLWAUA, seg_pos={0 0}, upload=337969]: Long tail cancellation due to optimal threshold Dec 21 16:33:18.061034645 (385h4m17.93211769s) [noise=true , address=145.79.227.38:4978 , piece_id=MLRTOXRTPJX55KIMLBW3YL4PGB3TAIFQNYALNCRY4WSGN4Q5ZL6Q, seg_pos={0 0}, upload=337969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 21 16:33:18.061040637 (385h4m17.932123672s) [noise=true , address=145.79.227.38:4978 , piece_id=MLRTOXRTPJX55KIMLBW3YL4PGB3TAIFQNYALNCRY4WSGN4Q5ZL6Q, seg_pos={0 0}, upload=337969]: Long tail cancellation due to optimal threshold Dec 21 16:33:18.061056998 (385h4m17.932140043s) [seg_pos={0 0}, upload=337969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:33:18.061131580 (385h4m17.932214615s) [upload=337969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:33:18.061138082 (385h4m17.932221127s) [upload=337969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:33:18.385313294 (385h4m18.256396329s) [upload=337969]: Done uploading segments... Dec 21 16:33:18.385332340 (385h4m18.256415375s) [upload=337969]: Upload finished. err: <nil> Duration: 9.145776834s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)