Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X...X.....................X.............X..................] duration:
3.464392519s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:42:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.464419932s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
4.89778263s trace:
https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3
FAIL (2025-12-08 09:42:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.897794993s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....X........................X.............................] duration:
5.80315927s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:42:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.803195177s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m23.767493069s trace:
https://jaeger.dev.storj.io/trace/11531c6fb6b54bc3
FAIL (2025-12-08 09:42:09 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1m23.767518179s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X...X.....................X.............X.................] duration:
3.563263607s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:42:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.563295478s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X.....X................] duration:
6.365514703s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:41:57 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.365567985s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......X........................X............................] duration:
5.811745138s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:41:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.81177234s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
6.894370146s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:41:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.894400764s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
471.272562ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:41:24 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 471.304614ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X.....X...............] duration:
6.289120516s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:41:21 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.289150132s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X...X.....................X.............X................] duration:
3.399749671s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:41:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.399789718s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
7.72665237s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:41:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 7.726909911s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
474.256394ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:41:08 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 474.311538ms
core
satellite
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.778553647s trace:
https://jaeger.dev.storj.io/trace/40c2b61890d7290
FAIL (2025-12-08 09:41:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.778605536s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
8.322149851s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:40:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.32217535s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......X........................X...........................] duration:
5.821049197s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:40:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.82109735s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
473.040367ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:40:41 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 473.077328ms
core
satellite
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....X...X.....................X.............X...............] duration:
3.51878s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:40:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.518807904s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X.....X..............] duration:
5.995388467s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:40:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.995422682s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....X...X.....................X.............X..............] duration:
3.547501223s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:40:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.547531371s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
8.074222251s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:40:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.074251707s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.977114162s trace:
https://jaeger.dev.storj.io/trace/40c2b61890d7290
FAIL (2025-12-08 09:39:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.977141064s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........X........................X..........................] duration:
5.830309901s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:39:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.830341952s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
471.088111ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:39:40 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 471.110233ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
7.927896791s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:39:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 7.928038101s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......X...X.....................X.............X.............] duration:
3.375727329s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:39:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.37575375s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X.....X.............] duration:
6.007523153s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:39:21 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.00755786s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X.....X............] duration:
6.064717644s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:39:04 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.0647415s
core
uplink
us1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
476.677825ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:38:57 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 476.703122ms
core
satellite
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.9643265s trace:
https://jaeger.dev.storj.io/trace/40c2b61890d7290
FAIL (2025-12-08 09:38:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.964354172s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X........................X.........................] duration:
5.753085463s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:38:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.753122464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
7.769131458s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:38:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 7.769162667s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
474.047727ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:38:35 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 474.076531ms
core
satellite
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......X...X.....................X.............X............] duration:
3.493354536s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:38:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.493369696s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
8.648999025s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:38:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.649029734s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........X........................X........................] duration:
5.762102243s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:38:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.76213131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X.....X...........] duration:
5.896282139s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:38:13 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.896312575s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........X...X.....................X.............X...........] duration:
3.537183364s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:38:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.537213722s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X.....X..........] duration:
6.184795761s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:37:52 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.184827593s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
7.97624299s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:37:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 7.976277816s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.844511479s trace:
https://jaeger.dev.storj.io/trace/40c2b61890d7290
FAIL (2025-12-08 09:37:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.844544291s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
468.980559ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:37:46 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 469.013121ms
core
satellite
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.096663144s trace:
https://jaeger.dev.storj.io/trace/40c2b61890d7290
FAIL (2025-12-08 09:37:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.096679487s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X.....X.........] duration:
6.191539286s trace:
https://jaeger.dev.storj.io/trace/576b3494e37dde53
FAIL (2025-12-08 09:37:17 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.19161456s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X........................X.......................] duration:
5.878809554s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:37:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.878863809s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
8.523487641s trace:
https://jaeger.dev.storj.io/trace/34a22ba17362ff8c
FAIL (2025-12-08 09:37:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 09:27:22.965855471 (65h58m22.836938516s) [upload=57851]: Starting upload
Dec 8 09:27:22.965932157 (65h58m22.837015192s) [upload=57851]: Done starting upload
Dec 8 09:27:22.965997491 (65h58m22.837080536s) [upload=57851]: Uploading segments...
Dec 8 09:27:22.966181472 (65h58m22.837264507s) [upload=57851]: Got next segment. Inline: false
Dec 8 09:27:22.966299707 (65h58m22.837382752s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 09:27:22.966306600 (65h58m22.837389645s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 09:27:24.179054828 (65h58m24.050137873s) [seg_pos={0 0}, upload=57851]: Begin upload segment...
Dec 8 09:27:24.179107689 (65h58m24.050190734s) [seg_pos={0 0}, upload=57851]: Done begin upload segment.
Dec 8 09:27:24.179146532 (65h58m24.050229567s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179151913 (65h58m24.050234949s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179194124 (65h58m24.050277159s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179587963 (65h58m24.050670998s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.179727419 (65h58m24.050810454s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180097414 (65h58m24.051180459s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180443633 (65h58m24.051526678s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180654274 (65h58m24.051737309s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180711512 (65h58m24.051794547s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.180970005 (65h58m24.05205304s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181347234 (65h58m24.052430269s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181605515 (65h58m24.05268855s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181660260 (65h58m24.052743305s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.181879477 (65h58m24.052962522s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182137327 (65h58m24.053220362s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:24.182246075 (65h58m24.05332911s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Uploading piece...
Dec 8 09:27:26.020756673 (65h58m25.891839708s) [upload=57851]: Next returned nil segment
Dec 8 09:27:26.020777182 (65h58m25.891860217s) [upload=57851]: Waiting for error group managing segments...
Dec 8 09:27:26.311385284 (65h58m26.182468319s) [noise=true , address=145.79.227.43:4159 , piece_id=JN6LZP66NGJTKQFHOH7HJWSNLUJTHMMSTD6EB3IHXVDITSTJYVLA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315067335 (65h58m26.18615037s) [noise=true , address=145.79.227.39:4014 , piece_id=LYKKLPY66C4746EVJMLIBBKIDRG34I6Y3O52VQLCRERD4A2N6EZQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.315543612 (65h58m26.186626647s) [noise=true , address=145.79.227.21:4474 , piece_id=YRTFHUSALJPNNWKS7VIDAODLDQAJGIEIDMUZMSAH6Q2MXDCJCWMQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.321177260 (65h58m26.192260305s) [noise=true , address=145.79.227.28:4727 , piece_id=6PYUF32FKVLNKFBX2BE33JORRZ4BQ24A6GJBCOPTZHAIM7ZPRPHQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.554771174 (65h58m26.425854209s) [noise=true , address=145.79.227.18:4007 , piece_id=NUDSZGS3U4XOXWJJ3C6EGFGYD3QJZ43THTDAVNWPDSU6MFFDSABQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.566832932 (65h58m26.437915967s) [noise=true , address=145.79.227.26:4655 , piece_id=7MZFMW6JBMEGHIHKWDOMXKJHE7T3AWNXPUJ7E4VWXZJ7Y2M5YQGA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.570956635 (65h58m26.44203968s) [noise=true , address=145.79.227.31:4799 , piece_id=3ZRXUTYHNJ2UA72WDD645D7I34E63MYNSJNATOVP7NEFJJZBCFBA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.576588048 (65h58m26.447671083s) [noise=true , address=145.79.227.42:4122 , piece_id=N7S6CX33FQRU22NDJFQQEEDLWXWQ2YXWNHHH2YOS62ZHGFZ3SE4A, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.604283045 (65h58m26.47536609s) [noise=true , address=145.79.227.26:4654 , piece_id=AUECLGQ4RAM2SQY6MXF4P5CPHZTHIWURGGJFSLBGBDBPTKOGMTYA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:26.630913405 (65h58m26.50199644s) [noise=true , address=77.90.31.11:4554 , piece_id=LA7TDZ362BG6J5WYM6OJ3RTIZRIRQAUBMS5A5IKMM765PHNGNNJQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.060541321 (65h58m26.931624366s) [noise=true , address=145.79.227.41:4085 , piece_id=GTRBEFJRX6VYDULUEYY6YJOW3W7MFAZBETOXHQGPQYK6KA6IXSMA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.132302570 (65h58m27.003385615s) [noise=true , address=145.79.227.42:4121 , piece_id=A6CC3S4QVFUURD4JXGI4ORYGG3S4XYUWEEOIGGG7FCBC36FDOULA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.148978365 (65h58m27.02006141s) [noise=true , address=145.79.227.31:4797 , piece_id=XNCM2JNSPK3ITQITZON4TJ5CVMQKDGOLHFQJO7ZGY6MEQLRE6ROA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149003803 (65h58m27.020086848s) [seg_pos={0 0}, upload=57851]: Segment reached optimal threshold of 13 pieces.
Dec 8 09:27:27.149304055 (65h58m27.0203871s) [noise=true , address=145.79.227.22:4510 , piece_id=IIVSY4NBQXYBBRLDBN7CE4MITPXAMJRT2T6APTURUKA6STNQEXYQ, seg_pos={0 0}, upload=57851]: Done uploading piece. err: <nil>
Dec 8 09:27:27.149440905 (65h58m27.02052395s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149451325 (65h58m27.02053437s) [noise=true , address=145.79.227.16:4912 , piece_id=ONPJOB7B5IYYWAOV36UPATA5AZ7IEEBUQABNK7M6CUDHD3HPVGOA, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149571655 (65h58m27.02065469s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 8 09:27:27.149578267 (65h58m27.020661302s) [noise=true , address=145.79.227.34:4907 , piece_id=DHZWSPJMNLFXJ3AO2PRTNAHRS3NC6Q4QBY2OCR5SQWOEMPH4HO4Q, seg_pos={0 0}, upload=57851]: Long tail cancellation due to optimal threshold
Dec 8 09:27:27.149594357 (65h58m27.020677392s) [seg_pos={0 0}, upload=57851]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 8 09:27:27.149665152 (65h58m27.020748187s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 09:27:27.149669590 (65h58m27.020752625s) [upload=57851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 09:29:06.190387981 (66h0m6.061471016s) [upload=57851]: Done uploading segments...
Dec 8 09:29:06.190472822 (66h0m6.061555857s) [upload=57851]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.52351841s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
475.837284ms remote:
"tcp:35.207.121.91:443" target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/56d89704e7febe70
FAIL (2025-12-08 09:37:05 UTC)
orbiter warning: Get "https://eu1.storj.io/login": EOF
Duration: 475.870719ms
core
satellite
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........X...X.....................X.............X..........] duration:
3.492086039s trace:
https://jaeger.dev.storj.io/trace/6a116075ee2153e5
FAIL (2025-12-08 09:36:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.492126566s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.969204805s trace:
https://jaeger.dev.storj.io/trace/40c2b61890d7290
FAIL (2025-12-08 09:36:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.969233219s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X........................X......................] duration:
5.670858302s trace:
https://jaeger.dev.storj.io/trace/56651b65c9cab08a
FAIL (2025-12-08 09:36:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.670885775s
core
uplink
us-select-1
us1-placement-20
|