Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 514985 reports (Page 5397 of 10300)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................X...X..X.......................] duration: 3.946396892s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:19:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.946738985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X...X..X......................] duration: 4.001021119s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:18:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.001054142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................X...X..X.....................] duration: 3.946576536s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.946616974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X...............................X...X..X....................] duration: 5.000785057s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:17:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000817108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X...............................X...X..X...................] duration: 3.98691545s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:16:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.986960576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X...............................X...X..X..................] duration: 4.014848042s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:15:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.014899359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X...............................X...X..X.................] duration: 3.965923093s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.965951829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X...............................X...X..X................] duration: 3.926369697s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:13:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.926397279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X...............................X...X..X...............] duration: 4.039923561s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:13:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.039960871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X...............................X...X..X..............] duration: 3.944807719s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:12:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.944839418s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X...............................X...X..X.............] duration: 4.155320449s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:12:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.155350957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X...............................X...X..X............] duration: 3.970733055s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:11:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.970763543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.820560847s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:11:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.820596847s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X...............................X...X..X...........] duration: 3.983882886s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:11:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.983903668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.339697951s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:10:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.339828279s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.........X...............................X...X..X..........] duration: 5.001462368s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:10:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001490843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.836122954s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:09:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.836155107s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.........X...............................X...X..X.........] duration: 4.039005749s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:09:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.039031969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.804793633s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.804837766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.........X...............................X...X..X........] duration: 3.977127021s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:08:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.97715786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.644488347s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:07:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.64451676s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X.........X...............................X...X..X.......] duration: 3.972459564s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:07:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.972493868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.833038478s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.833067594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.223726788s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:06:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.223756954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.447692627s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:06:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.44772575s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X.........X...............................X...X..X......] duration: 3.976900608s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:06:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.976930895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.121718651s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:05:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.121737466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.545505912s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:05:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.545533906s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X.........X...............................X...X..X.....] duration: 4.151426492s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:05:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.151460768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.163056258s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:04:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.163231392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......X.........X...............................X...X..X....] duration: 5.000789217s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:04:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000907001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 8.874120322s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:04:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 8.87414485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.085042331s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.085071358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.969566533s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:03:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.96959625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......X.........X...............................X...X..X...] duration: 4.003215273s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:03:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.003374557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.102694271s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:03:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.102723397s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X.........X...............................X...X..X..] duration: 4.196206221s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:02:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.196239043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 9.981590762s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 9.981638935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.652484482s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.652507526s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........X.........X...............................X...X..X.] duration: 3.993709585s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:01:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.993747869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X.........X...............................X...X..X] duration: 5.001068568s trace: https://jaeger.dev.storj.io/trace/6dc5a643275b2649 FAIL (2025-11-11 18:01:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001125947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.083937494s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:01:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.083970908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.33326851s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:00:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.333332244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.186632599s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:00:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.186665772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X.........X...............................X...X..] duration: 3.952479223s trace: https://jaeger.dev.storj.io/trace/444bf3f58671b75 FAIL (2025-11-11 18:00:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.952510312s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.912711404s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 18:00:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.912744558s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X.........X...............................X...X.] duration: 4.221603122s trace: https://jaeger.dev.storj.io/trace/444bf3f58671b75 FAIL (2025-11-11 18:00:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.22171209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.835621963s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 18:00:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.835650697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.789040736s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:59:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.789066024s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.360570061s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.360614587s core uplink us-select-1 us1-placement-20

1
...
5395
5396
5397
5398
5399
...
10300