Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 860534 reports (Page 11766 of 17211)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X....X........................X..............X...........] duration: 5.793435202s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:14:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.793465009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X.............................X....X.....X..............] duration: 3.571516752s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:14:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571550546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 4.856490794s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:13:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.856516253s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.............................X....X.....X.............] duration: 3.473947064s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:13:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473974055s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X....X........................X..............X..........] duration: 5.960048117s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:13:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.960074086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 6.006652561s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:13:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.006682137s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.978614079s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:13:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.978643335s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X....X........................X..............X.........] duration: 5.972792385s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:13:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.972808695s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X.............................X....X.....X............] duration: 3.486188767s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:12:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486264732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 4.903354754s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:12:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.90338445s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 5.865737103s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:12:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.865768994s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X....X........................X..............X........] duration: 5.907703053s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:12:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.907750383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.925037292s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:11:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.925065746s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X.............................X....X.....X...........] duration: 3.459995962s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:11:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460021841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 6.270235087s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:11:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.270262428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X....X........................X..............X.......] duration: 5.837303872s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:11:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.837361983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X.............................X....X.....X..........] duration: 3.407915933s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:11:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407941071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.03058981s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:11:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.030620027s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X....X........................X..............X......] duration: 5.90008156s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:11:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.900109905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X.............................X....X.....X.........] duration: 3.498941652s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498970576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 4.861444427s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:10:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.861479434s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 6.758264177s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:10:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.758289525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X....X........................X..............X.....] duration: 6.050574324s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:10:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.050599092s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X.............................X....X.....X........] duration: 3.530951895s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:10:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53109666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.287969185s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:10:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.288002358s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.10123447s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:09:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.101264386s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 6.552448709s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:09:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.55247078s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X....X........................X..............X....] duration: 5.793136708s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:09:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.793164452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X.............................X....X.....X.......] duration: 3.478056333s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:09:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478199855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 6.26889234s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:09:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.26896649s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X....X........................X..............X...] duration: 5.823084105s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:09:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.823105867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 4.915422854s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:08:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.915445396s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X.............................X....X.....X......] duration: 3.380887165s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:08:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.380915598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.936817241s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:08:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.936846518s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X....X........................X..............X..] duration: 5.924787897s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:08:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.924814207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X.............................X....X.....X.....] duration: 3.451415643s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:07:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45144027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.143627162s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:07:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.143656799s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 5.835041464s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:07:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.835072022s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X....X........................X..............X.] duration: 5.922632291s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:07:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.922646408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 13.870879367s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.870910606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X.............X....X........................X..............X] duration: 1m0.000736104s trace: https://jaeger.dev.storj.io/trace/79dc68a558cc00ae FAIL (2025-12-08 01:07:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000753136s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X.............................X....X.....X....] duration: 3.475221604s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:06:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475321355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.217783399s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:06:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.217803567s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 5.892111975s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:06:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.892146169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.907525574s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:05:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.90755404s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.049858421s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:05:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.049886936s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............X.............................X....X.....X...] duration: 3.448487138s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:05:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44851962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 5.742105116s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.742133459s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X....X........................X..............] duration: 6.093029146s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 01:05:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.093053414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X.............................X....X.....X..] duration: 3.545211667s trace: https://jaeger.dev.storj.io/trace/79b8d274a6c2d17e FAIL (2025-12-08 01:05:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545241704s core uplink us-select-1 us1-placement-20

1
...
11764
11765
11766
11767
11768
...
17211