Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 860524 reports (Page 11767 of 17211)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 5.742293926s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:02:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.742323833s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X.............................X....X.....] duration: 3.360760811s trace: https://jaeger.dev.storj.io/trace/6e0f7fa8bfd08dee FAIL (2025-12-08 01:02:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.360791239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X...X.............X....X........................X..........] duration: 5.674524954s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 01:02:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.674553667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X.............................X....X....] duration: 3.461656464s trace: https://jaeger.dev.storj.io/trace/6e0f7fa8bfd08dee FAIL (2025-12-08 01:01:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461674198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 4.97515384s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:01:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.975207873s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 5.857183906s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:01:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.857210798s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.399822142s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:01:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.399845667s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 6.187536935s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 01:01:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.187586409s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X...X.............X....X........................X.........] duration: 5.928117936s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 01:01:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.928148304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X.............................X....X...] duration: 3.504012889s trace: https://jaeger.dev.storj.io/trace/6e0f7fa8bfd08dee FAIL (2025-12-08 01:00:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50404005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.068274529s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:00:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.068299787s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X.............................X....X..] duration: 3.475985047s trace: https://jaeger.dev.storj.io/trace/6e0f7fa8bfd08dee FAIL (2025-12-08 01:00:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476104894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.967364597s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:00:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.967399453s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X...X.............X....X........................X........] duration: 5.771475741s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 01:00:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.771495679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.897672052s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 01:00:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.89771845s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.713777627s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:59:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.713807022s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X.............................X....X.] duration: 3.466160058s trace: https://jaeger.dev.storj.io/trace/6e0f7fa8bfd08dee FAIL (2025-12-08 00:59:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466188563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 5.896712415s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:59:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.896838705s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 4.952556342s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:59:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.952587379s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X.............................X....X] duration: 5.001605999s trace: https://jaeger.dev.storj.io/trace/6e0f7fa8bfd08dee FAIL (2025-12-08 00:59:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001634014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 4.750868453s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:58:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.750884023s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X...X.............X....X........................X.......] duration: 5.761142447s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:58:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.761176142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X...X.............X....X........................X......] duration: 5.773276473s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:58:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.773304496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.007883213s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:58:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.007909654s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 6.191395518s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:58:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.191423251s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X...X.............X....X........................X.....] duration: 5.898585602s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:58:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.898617411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X.............................X....] duration: 3.404486461s trace: https://jaeger.dev.storj.io/trace/2fc0cdf6a5f5c519 FAIL (2025-12-08 00:57:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.404519614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 4.933666631s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:57:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.933693141s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X.............X....X........................X....] duration: 5.715690439s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:57:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.715717981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.965177856s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:57:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.965200989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 5.663786856s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:57:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.663842521s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X.............................X...] duration: 3.540499694s trace: https://jaeger.dev.storj.io/trace/2fc0cdf6a5f5c519 FAIL (2025-12-08 00:57:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540531776s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X...X.............X....X........................X...] duration: 5.904032975s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:57:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.904066259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 4.999671918s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:57:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.999702986s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 5.847209019s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:56:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.847234227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........X...X.............X....X........................X..] duration: 6.01665168s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:56:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.016720321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X.............................X..] duration: 3.494661551s trace: https://jaeger.dev.storj.io/trace/2fc0cdf6a5f5c519 FAIL (2025-12-08 00:56:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494706346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.022595062s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:56:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.022626282s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 5.656965577s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:56:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.656995663s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........X...X.............X....X........................X.] duration: 5.838023422s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:55:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.83803793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 5.899911296s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:55:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.899940713s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X..........X...X.............X....X........................X] duration: 1m0.001564058s trace: https://jaeger.dev.storj.io/trace/27f8e2bf98dcc828 FAIL (2025-12-08 00:55:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001592251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X.............................X.] duration: 3.562130875s trace: https://jaeger.dev.storj.io/trace/2fc0cdf6a5f5c519 FAIL (2025-12-08 00:55:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562160812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.939866616s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:55:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.939896123s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 45.371638089s trace: https://jaeger.dev.storj.io/trace/5d6e0eb337ed2afd FAIL (2025-12-08 00:54:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 00:54:14.374317674 (57h25m14.245400709s) [upload=50456]: Starting upload Dec 8 00:54:14.374418425 (57h25m14.24550147s) [upload=50456]: Done starting upload Dec 8 00:54:14.374552591 (57h25m14.245635636s) [upload=50456]: Uploading segments... Dec 8 00:54:14.374678130 (57h25m14.245761165s) [upload=50456]: Got next segment. Inline: false Dec 8 00:54:14.374744747 (57h25m14.245827782s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 00:54:14.374749937 (57h25m14.245832972s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 00:54:14.626337150 (57h25m14.497420225s) [seg_pos={0 0}, upload=50456]: Begin upload segment... Dec 8 00:54:14.626389330 (57h25m14.497472375s) [seg_pos={0 0}, upload=50456]: Done begin upload segment. Dec 8 00:54:14.626418335 (57h25m14.49750138s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626447320 (57h25m14.497530365s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.626492506 (57h25m14.497575542s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627006514 (57h25m14.49808956s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.627504523 (57h25m14.498587558s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628021658 (57h25m14.499104693s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628353319 (57h25m14.499436364s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.628743903 (57h25m14.499826948s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.629144686 (57h25m14.500227722s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.630582614 (57h25m14.501665659s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.632070618 (57h25m14.503153693s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.633430617 (57h25m14.504513653s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.635064158 (57h25m14.506147193s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.636514670 (57h25m14.507597706s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.637229231 (57h25m14.508312276s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:14.639163083 (57h25m14.510246118s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Uploading piece... Dec 8 00:54:16.296771158 (57h25m16.167854203s) [upload=50456]: Next returned nil segment Dec 8 00:54:16.296801344 (57h25m16.167884379s) [upload=50456]: Waiting for error group managing segments... Dec 8 00:54:16.622222430 (57h25m16.493305465s) [noise=true , address=145.79.227.41:4085 , piece_id=6FCZPGMJT3T2QDPZ2ESUJQTG3FB2EZMHKO2L4SPY6APDLCHID4EQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.624087531 (57h25m16.495170566s) [noise=true , address=145.79.227.23:4546 , piece_id=5D6S4KGPH4DQL24NFSUD7O7Q3IKQRFEFUVXMUVCLDANZWH7MRGYQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.625681907 (57h25m16.496764952s) [noise=true , address=145.79.227.29:4941 , piece_id=C46ZGHZOBISCTWQNY7WOHAT46Y5HGYD45LMVEIMMFGJLTJTNVQ3Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.626194312 (57h25m16.497277357s) [noise=true , address=145.79.227.40:4049 , piece_id=46YGFJZCCYJ3JO6W3FSVIPSVQDSAXNSKAWWE2ENZ34J6U3GJEEIQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.630827635 (57h25m16.50191068s) [noise=true , address=145.79.227.20:4439 , piece_id=GT3ZV6ZFP5LYGX6NX2IJXZYN3RFUEPTLTTZNZPQ7TIOSVTAHX5JQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.631842407 (57h25m16.502925442s) [noise=true , address=145.79.227.22:4509 , piece_id=FJXWNUK7725XGM6SHRNXFHCHQ3O5PRAPM3JK7CCN6VTOMXDSYQCQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.635918189 (57h25m16.507001234s) [noise=true , address=145.79.227.21:4473 , piece_id=P6QLQGROOYAHLIUNQC26C7R77M4ABQI6KW346S47THZ2G5HR36JA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.636120493 (57h25m16.507203538s) [noise=true , address=145.79.227.41:4087 , piece_id=ZMNGLBE5TNLXYSLJDBDQ3SPFZTZCUQK22H45TJUEXEG5CPHF5NKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.641936498 (57h25m16.513019543s) [noise=true , address=145.79.227.8:4329 , piece_id=7PLKWJPS7HN6U64FZ4LX6GXLQJYAYEDFYFEQVB4KO3HBKN2CTFOA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.647395623 (57h25m16.518478668s) [noise=true , address=145.79.227.23:4547 , piece_id=HCODYTNDOH2HU2BDNCQ4EXLNNGP6XUF3YYYP7NXISMX5IV4ANNPQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655023898 (57h25m16.526106933s) [noise=true , address=145.79.227.20:4438 , piece_id=HZEAVSXCX3WAAWKZ4BXHYKBM7KJE5G4SI35ZFHHFXZ3TYVIGMCKA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.655082951 (57h25m16.526165986s) [noise=true , address=145.79.227.32:4836 , piece_id=XCUCE2Y25UOOUNMSMQ572UM32OX5VMYN6EEQAIWLSCJQTNHZUX6Q, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666773382 (57h25m16.537856427s) [noise=true , address=145.79.227.16:4912 , piece_id=7UQG43MODL4QTMVH2XFINI3BBG3PX7F6DQ3DKKWKV33JSC5KFL6A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: <nil> Dec 8 00:54:16.666801304 (57h25m16.537884339s) [seg_pos={0 0}, upload=50456]: Segment reached optimal threshold of 13 pieces. Dec 8 00:54:16.667100134 (57h25m16.538183189s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667111916 (57h25m16.538194971s) [noise=true , address=145.79.227.24:4583 , piece_id=UYB3XUZG4SSQBMIQ54SO3YNMKCVNIBUERHAVJULYUSZVIBP7F2SA, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667241873 (57h25m16.538324919s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667262082 (57h25m16.538345127s) [noise=true , address=77.90.31.14:4661 , piece_id=J6X4STAGF33JIXZK5AFWUFZZU4ZF6HR7YOGFYDRZ5OXBT2Q65E2A, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667521746 (57h25m16.538604821s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Dec 8 00:54:16.667533308 (57h25m16.538616353s) [noise=true , address=77.90.31.18:4807 , piece_id=5READPCTGBO7TQQ77ICPBKGO3N77JSCAJIO7AP4IIWLRXVFVNKZQ, seg_pos={0 0}, upload=50456]: Long tail cancellation due to optimal threshold Dec 8 00:54:16.667560530 (57h25m16.538643565s) [seg_pos={0 0}, upload=50456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 00:54:16.667743669 (57h25m16.538826714s) [upload=50456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 00:54:16.667753869 (57h25m16.538836914s) [upload=50456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 00:54:58.351569004 (57h25m58.222652049s) [upload=50456]: Done uploading segments... Dec 8 00:54:58.351739419 (57h25m58.222822454s) [upload=50456]: Upload finished. err: metaclient: context deadline exceeded Duration: 45.37165994s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X............................X.............................X] duration: 5.001109202s trace: https://jaeger.dev.storj.io/trace/2fc0cdf6a5f5c519 FAIL (2025-12-08 00:54:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001130514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.08128032s trace: https://jaeger.dev.storj.io/trace/76defe050ad1aa06 FAIL (2025-12-08 00:54:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.081307783s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001422382s trace: https://jaeger.dev.storj.io/trace/77af65d5fc027a08 FAIL (2025-12-08 00:54:08 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 30.001444172s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X..........X...X.............X....X........................] duration: 5.900164327s trace: https://jaeger.dev.storj.io/trace/200dccf7ecc0a24e FAIL (2025-12-08 00:54:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.900195636s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X............................X.............................] duration: 4.334770793s trace: https://jaeger.dev.storj.io/trace/2c6e5f4ecf2a6935 FAIL (2025-12-08 00:53:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.334798998s core uplink us-select-1 us1-placement-20

1
...
11765
11766
11767
11768
11769
...
17211