Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 869193 reports (Page 11619 of 17384)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.03031767s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:13:38 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.030344901s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X...X.......................X............XX.] duration: 3.462295418s trace: https://jaeger.dev.storj.io/trace/2a245e66c865945e FAIL (2025-12-09 11:13:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462329534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.75002672s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:12:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.750066505s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.951052298s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:12:32 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.951076593s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X...X.X.......X..X........X.....X...................] duration: 5.828384228s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:12:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.828510198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X...X.......................X............XX] duration: 5.000502441s trace: https://jaeger.dev.storj.io/trace/2a245e66c865945e FAIL (2025-12-09 11:12:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000523021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.441443813s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:11:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.441475683s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................X...X.......................X............X] duration: 5.000534665s trace: https://jaeger.dev.storj.io/trace/59213b01a8e2f1ff FAIL (2025-12-09 11:11:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000574852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 4.021896484s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:11:33 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 4.021928194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X.X.......X..X........X.....X..................] duration: 5.872874592s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:11:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.872902865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.788728001s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:11:15 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.788756877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.974020002s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:10:57 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.974065497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.254221898s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:10:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.254238468s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X...X.......................X............] duration: 3.580104689s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:10:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.580130257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 15m10.630720902s trace: https://jaeger.dev.storj.io/trace/4f140252d772775e FAIL (2025-12-09 11:10:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 15m10.630747001s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.080994711s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:10:20 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.081028815s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..........X...X.X.......X..X........X.....X.................] duration: 5.842272579s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:10:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.842369613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X...X.......................X...........] duration: 3.509120115s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:09:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509149821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.83201358s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:09:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.832043497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000841022s trace: https://jaeger.dev.storj.io/trace/e55dae4c17f764 FAIL (2025-12-09 11:09:43 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 30.000864216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 7.071472544s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:09:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.071502643s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X...X.......................X..........] duration: 3.4982743s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:09:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498313455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...........X...X.X.......X..X........X.....X................] duration: 5.928950103s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:08:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.928986252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.601966607s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:08:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.602021572s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [............X...X.X.......X..X........X.....X...............] duration: 5.694597629s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:08:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.694618429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X...X.......................X.........] duration: 3.501629739s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:08:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501660758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.109943256s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:07:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.109966141s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.............X...X.X.......X..X........X.....X..............] duration: 5.948240004s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:07:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.948297232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.152713715s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:07:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.152745977s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X...X.......................X........] duration: 3.404923159s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:07:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40493991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..............X...X.X.......X..X........X.....X.............] duration: 5.698494029s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:07:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.698523364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.878153044s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:06:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.878188301s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X...X.......................X.......] duration: 3.598432649s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:06:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598461704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...............X...X.X.......X..X........X.....X............] duration: 6.09600744s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:06:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.096043871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.543178091s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:05:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.543206666s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [................X...X.X.......X..X........X.....X...........] duration: 5.657886482s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:05:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.657919384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.65537353s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:05:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.65539995s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X...X.......................X......] duration: 3.550179265s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:05:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.55020788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.................X...X.X.......X..X........X.....X..........] duration: 5.989175624s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:04:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.989204719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X...X.......................X.....] duration: 3.511138545s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:04:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511168452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.227486014s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:03:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.227678931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X...X.......................X....] duration: 3.484010527s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:03:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484090059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..................X...X.X.......X..X........X.....X.........] duration: 5.784979065s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:03:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.785005015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...................X...X.X.......X..X........X.....X........] duration: 5.874162215s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:02:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.874205927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 12.114511343s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:02:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.114540961s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X...X.......................X...] duration: 3.524948803s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:02:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524971835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....................X...X.X.......X..X........X.....X.......] duration: 5.841948454s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:01:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.841976907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.841820251s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 11:01:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841849508s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X............................X...X.......................X..] duration: 3.515235226s trace: https://jaeger.dev.storj.io/trace/1787ff005d88f7e7 FAIL (2025-12-09 11:01:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515258621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....................X...X.X.......X..X........X.....X......] duration: 5.846143933s trace: https://jaeger.dev.storj.io/trace/5a658a11cb4dd62d FAIL (2025-12-09 11:01:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.846190421s core uplink us-select-1 us1-placement-20

1
...
11617
11618
11619
11620
11621
...
17384