Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....X...........XX..X.......X...X..................X.......] duration: 6.003655823s trace: https://jaeger.dev.storj.io/trace/13c3e87d490685f0 FAIL (2025-12-09 08:01:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.00375363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....................X...................] duration: 5.922949613s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:01:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.922985922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.311350699s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 08:01:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.311388281s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.987637905s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 08:01:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.987676097s core uplink us1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 461.772647ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 08:00:57 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 461.8112ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 2.121020901s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 08:00:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.121059244s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 5.890132625s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 08:00:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.890164327s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.938030142s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 08:00:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.938062574s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [..........XX..X.............................................] duration: 684.67122ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 08:00:44 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 684.793712ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....................X..................] duration: 5.743785215s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:00:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.743816815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.322014098s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 08:00:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.322050909s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......X...........XX..X.......X...X..................X......] duration: 5.945824779s trace: https://jaeger.dev.storj.io/trace/13c3e87d490685f0 FAIL (2025-12-09 08:00:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.945874222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.237562494s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 08:00:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.237596969s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..............X.XX.XX..X..................X.................] duration: 3.471094892s trace: https://jaeger.dev.storj.io/trace/610b5afc07f67951 FAIL (2025-12-09 08:00:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471112126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XXX.X.............................................] duration: 5.768311196s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 08:00:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.768343759s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 5.838772923s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 08:00:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.838804493s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [...........XX..X............................................] duration: 694.877344ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 08:00:20 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 694.908813ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.971801177s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 08:00:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.971839971s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.05280889s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 08:00:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.052856521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.968495362s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:00:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.96854179s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 2.509913669s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 08:00:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.510011906s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...............X.XX.XX..X..................X................] duration: 3.388339776s trace: https://jaeger.dev.storj.io/trace/610b5afc07f67951 FAIL (2025-12-09 07:59:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.38836738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.928353255s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:59:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.928380677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.91480757s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 07:59:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.914827738s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 434.883432ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:59:50 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 434.915374ms edge linksharing us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [............XX..X...........................................] duration: 685.376361ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:59:50 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 685.401259ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....................X.................] duration: 5.766970724s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 07:59:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.767018255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [................X.XX.XX..X..................X...............] duration: 3.542221057s trace: https://jaeger.dev.storj.io/trace/610b5afc07f67951 FAIL (2025-12-09 07:59:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542261894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 2.192561749s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:59:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.192586346s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.86484463s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:59:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.86486544s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.044277127s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:59:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.044308076s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......X...........XX..X.......X...X..................X.....] duration: 5.891213162s trace: https://jaeger.dev.storj.io/trace/13c3e87d490685f0 FAIL (2025-12-09 07:59:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.891241335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....................X................] duration: 5.725532032s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 07:59:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.725562649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 6.179507688s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:59:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.179581739s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.962794838s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:59:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.962852999s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XXX.X............................................] duration: 5.736606199s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:59:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.736637087s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [.............XX..X..........................................] duration: 702.669418ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:59:22 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 702.694647ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.907585924s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 07:59:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.907620209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X...........XX..X.......X...X..................X....] duration: 5.637788482s trace: https://jaeger.dev.storj.io/trace/13c3e87d490685f0 FAIL (2025-12-09 07:59:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.638453949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 2.194956944s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:59:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.194985939s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 4.859809179s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:58:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.85984611s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.011674228s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:58:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.011695469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.936623352s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:58:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.936659561s core uplink us1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 448.371906ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:58:53 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 448.395132ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XXX.X...........................................] duration: 5.596732124s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:58:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.59676139s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X...........XX..X.......X...X..................X...] duration: 6.787949886s trace: https://jaeger.dev.storj.io/trace/13c3e87d490685f0 FAIL (2025-12-09 07:58:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.787978923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.918646847s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 07:58:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.918675562s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [..............XX..X.........................................] duration: 692.196458ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:58:39 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 692.232145ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.................X.XX.XX..X..................X..............] duration: 4.39240431s trace: https://jaeger.dev.storj.io/trace/610b5afc07f67951 FAIL (2025-12-09 07:58:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.392434598s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 6.133617339s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:58:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.133642316s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)