Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 853367 reports (Page 11857 of 17068)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.417699278s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:19:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417727582s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 400.430872ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6b3038bfdb17b2f5 FAIL (2025-11-24 00:19:03 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": EOF Duration: 400.45052ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.418191494s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:18:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.418217884s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 345.631716ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6b3038bfdb17b2f5 FAIL (2025-11-24 00:18:15 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": EOF Duration: 345.660151ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.52089043s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:17:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520916779s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 312.718264ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6b3038bfdb17b2f5 FAIL (2025-11-24 00:17:05 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": EOF Duration: 312.739674ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 86.34337ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6b3038bfdb17b2f5 FAIL (2025-11-24 00:16:49 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": EOF Duration: 86.362165ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.547219652s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:16:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547235032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.49139713s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:15:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491414693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.469531844s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:14:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469565218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.80193607s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:14:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.80196733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.367366586s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:12:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.367391854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.523064958s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:12:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523091458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.548112931s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:11:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548139994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001528268s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:10:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001543277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.800135901s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:05:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.800166749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.953019198s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:04:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.953049997s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.527828315s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 20:04:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527867579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.835299843s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:04:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.835330049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.916948851s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:03:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.916981013s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.516973107s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 20:03:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516998215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.44772448s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 20:02:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447795586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.908803162s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:02:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.908835715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.902953866s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:01:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.902980858s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.581154856s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 20:01:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.581190874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.018174484s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:01:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.018200564s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.485210354s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 20:00:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485237255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.022977068s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 20:00:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.023012525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.506657804s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:59:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506680689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.946239117s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:59:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.946266408s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.464974126s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:59:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465016537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.820722662s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:58:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.820750856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.52165438s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:58:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5217125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.965083447s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:57:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.965135677s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.37678823s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:57:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.376813067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.592217479s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:57:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592259791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.120534642s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:57:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.120556814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.793835208s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:56:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793861069s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.459209166s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:56:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459239193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.489033725s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:55:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489065045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.022035348s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:55:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.022061968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.788771562s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:55:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.788810987s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.404147725s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:55:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.404178074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.842520659s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:54:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.842542312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.947036144s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:54:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.947111067s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.510559075s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:54:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510588401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.116529455s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:53:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.116556697s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.538273344s trace: https://jaeger.dev.storj.io/trace/139543633a207e15 FAIL (2025-11-23 19:53:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538304283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.897423372s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:52:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.897456184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.909159382s trace: https://jaeger.dev.storj.io/trace/167024fcc8ed7afb FAIL (2025-11-23 19:52:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 23 19:21:12.380382025 (347h17m10.931852486s) [upload=305741]: Starting upload Nov 23 19:21:12.380465544 (347h17m10.931936005s) [upload=305741]: Done starting upload Nov 23 19:21:12.380580853 (347h17m10.932051314s) [upload=305741]: Uploading segments... Nov 23 19:21:12.380846970 (347h17m10.932317431s) [upload=305741]: Got next segment. Inline: false Nov 23 19:21:12.380921732 (347h17m10.932392193s) [upload=305741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 23 19:21:12.380929347 (347h17m10.932399808s) [upload=305741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 23 19:21:12.664451934 (347h17m11.215922395s) [seg_pos={0 0}, upload=305741]: Begin upload segment... Nov 23 19:21:12.664528089 (347h17m11.21599855s) [seg_pos={0 0}, upload=305741]: Done begin upload segment. Nov 23 19:21:12.664604424 (347h17m11.216074885s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.664625174 (347h17m11.216095645s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665427992 (347h17m11.216898453s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.665978079 (347h17m11.21744854s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666411886 (347h17m11.217882337s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.666825173 (347h17m11.218295634s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667383596 (347h17m11.218854057s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.667789198 (347h17m11.219259659s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668159413 (347h17m11.219629874s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.668642102 (347h17m11.220112563s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669030562 (347h17m11.220501023s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669415124 (347h17m11.220885585s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.669902773 (347h17m11.221373244s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670171214 (347h17m11.221641675s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670432120 (347h17m11.221902571s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:12.670723616 (347h17m11.222194097s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Uploading piece... Nov 23 19:21:15.138355395 (347h17m13.689825856s) [upload=305741]: Next returned nil segment Nov 23 19:21:15.138369381 (347h17m13.689839832s) [upload=305741]: Waiting for error group managing segments... Nov 23 19:21:15.834540725 (347h17m14.386011186s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6TQLPY7TRUVO25WHMJ7XP53FP7ZUDPETOZZ7ZPJNJPUTSA7563Q, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.071594267 (347h17m14.623064718s) [noise=true , address=77.90.31.18:4807 , piece_id=CIMELODYYCYHKSB4G23HNHTFY3VCBPLD4S635DT4BNKQXLBZU7HQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.315504416 (347h17m14.866974877s) [noise=true , address=45.91.116.6:4975 , piece_id=K7YWMW6PRTLDLJCCZUHK4WAR7GI4225BNZBQLGVQQALJZXIJKZNQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.358765994 (347h17m14.910236455s) [noise=true , address=77.90.31.13:4627 , piece_id=4KGNQNBBQLRDX2PYL4LZUTLF6ZUQZI36NHNDTXRUGKZOCPQZBCNA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:16.896643889 (347h17m15.44811434s) [noise=true , address=77.90.31.10:4518 , piece_id=RYLNF757RMW7N44EK6LALPY6HQDEGDXXEJ5ORYICZOK3DBYPJDUA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:17.922803819 (347h17m16.47427429s) [noise=true , address=145.79.227.24:4582 , piece_id=YJBB6WJW2IAVGGYGE4FSF25EABEL2TKOBJAJZ7NMJC5UBXBD26DA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:18.383941697 (347h17m16.935412158s) [noise=true , address=145.79.227.8:4329 , piece_id=E4BNJXC7GDU4M7EX3K24YVMG7DL757AEY4R7LDAE2B6NZLW7Q4EA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:19.790064386 (347h17m18.341534847s) [noise=true , address=145.79.227.22:4510 , piece_id=5BGA5U7FBAADZGOY7E5DU2NTANLVL6CZT65GFC2M3VRHFSOCB7BQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:21.570202896 (347h17m20.121673357s) [noise=true , address=145.79.227.25:4617 , piece_id=SMJDAS6M3V6TLENCN4TA4SXLZAG6EDF4GAPNTSHYZPDXESY2OJJA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:35.253110790 (347h17m33.804581241s) [noise=true , address=145.79.227.10:4402 , piece_id=DWJVGITQZFCFTUIKCB75H2XEKBG3JUJ4FGEH6U35QHULPXEVAF5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:37.912535507 (347h17m36.464005968s) [noise=true , address=145.79.227.20:4437 , piece_id=LDFWGZFUPBAND62VVI2LVZNUHBV2QC3BGUKEAKRM6MBTNYKYTYCA, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:38.093285897 (347h17m36.644756348s) [noise=true , address=145.79.227.21:4474 , piece_id=2EXYAVQIQZD3QHYH6ICUCX2GRNPY6E3FBLBTORDAQZUPZX7TSYLQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: <nil> Nov 23 19:21:41.573336882 (347h17m40.124807343s) [noise=true , address=145.79.227.27:4691 , piece_id=NCIOEPGYMSA6TDAPZ6BA4T6CUMDX7V3M7GS7WIGET6NW5WKSWDOQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: expected piece hash; context deadline exceeded; EOF Nov 23 19:21:41.573410883 (347h17m40.124881344s) [noise=true , address=145.79.227.11:4731 , piece_id=7TKZ2UDFTXLOKBEHTHQSOMNLLWWQZV2QTDQPNDIBD3ZW2SFUZM5A, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43488->145.79.227.11:4731: use of closed network connection Nov 23 19:21:41.573523708 (347h17m40.124994179s) [noise=true , address=145.79.227.13:4803 , piece_id=LASGHSPHVDNF3XNXSYYLRHAJWKIJSLVJ7KFKPOYYQYNSZ6E3STMQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38038->145.79.227.13:4803: use of closed network connection Nov 23 19:21:41.573682160 (347h17m40.125152631s) [noise=true , address=145.79.227.16:4910 , piece_id=RXRIAKBS5YCQ5SUNDGLCT6BXUBVTS427AJA47IVNA6JHCE54TYKQ, seg_pos={0 0}, upload=305741]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->145.79.227.16:4910: use of closed network connection Nov 23 19:21:41.573746962 (347h17m40.125217424s) [seg_pos={0 0}, upload=305741]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 23 19:21:41.816273035 (347h17m40.367743516s) [upload=305741]: Done uploading segments... Nov 23 19:21:41.816319855 (347h17m40.367790306s) [upload=305741]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.90918979s core uplink us-select-1

1
...
11855
11856
11857
11858
11859
...
17068