Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..X...X..XXX.X.XX...X...XX.............X...................] duration: 5.92219558s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:27:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.922227961s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............................X.........X.X......XX....X.....] duration: 5.001472006s trace: https://jaeger.dev.storj.io/trace/6892bd4d7f5b2cc8 FAIL (2025-11-21 12:26:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001656787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..X...X..XXX.X.XX...X...XX.............X..................] duration: 6.098384336s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:26:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.098492312s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............................X.........X.X......XX....X....] duration: 5.001926404s trace: https://jaeger.dev.storj.io/trace/6892bd4d7f5b2cc8 FAIL (2025-11-21 12:25:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001963284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X..X...X..XXX.X.XX...X...XX.............X.................] duration: 6.041339853s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:25:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.041368347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X..X...X..XXX.X.XX...X...XX.............X................] duration: 6.010561224s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:24:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.010612011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............................X.........X.X......XX....X...] duration: 4.835185044s trace: https://jaeger.dev.storj.io/trace/6892bd4d7f5b2cc8 FAIL (2025-11-21 12:24:37 UTC) orbiter warning: context deadline exceeded Duration: 4.835247403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.X..X...X..XXX.X.XX...X...XX.............X...............] duration: 6.010951157s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:24:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.010980362s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................................X.........X.X......XX....X..] duration: 4.324627344s trace: https://jaeger.dev.storj.io/trace/6892bd4d7f5b2cc8 FAIL (2025-11-21 12:24:03 UTC) orbiter warning: context deadline exceeded Duration: 4.32465652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X.X..X...X..XXX.X.XX...X...XX.............X..............] duration: 5.718503158s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:23:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.718536712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.................................X.........X.X......XX....X.] duration: 5.000804312s trace: https://jaeger.dev.storj.io/trace/6892bd4d7f5b2cc8 FAIL (2025-11-21 12:23:15 UTC) orbiter warning: context deadline exceeded Duration: 5.00082997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X.X..X...X..XXX.X.XX...X...XX.............X.............] duration: 6.034350742s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:23:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.03438688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X..X.X..X...X..XXX.X.XX...X...XX.............X............] duration: 5.95239776s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:22:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.952432195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..................................X.........X.X......XX....X] duration: 5.000941695s trace: https://jaeger.dev.storj.io/trace/6892bd4d7f5b2cc8 FAIL (2025-11-21 12:22:41 UTC) orbiter warning: context deadline exceeded Duration: 5.00096556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X..X.X..X...X..XXX.X.XX...X...XX.............X...........] duration: 6.005954112s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:22:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.005983306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................................X.........X.X......XX....] duration: 3.757452966s trace: https://jaeger.dev.storj.io/trace/1b1ff0c2189283a2 FAIL (2025-11-21 12:22:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.757483183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X..........] duration: 5.927285494s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:21:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.927316894s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....................................X.........X.X......XX...] duration: 5.001263986s trace: https://jaeger.dev.storj.io/trace/1b1ff0c2189283a2 FAIL (2025-11-21 12:21:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001317669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X.........] duration: 6.104630619s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:20:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.104661878s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X....................................X.........X.X......XX..] duration: 4.678825623s trace: https://jaeger.dev.storj.io/trace/1b1ff0c2189283a2 FAIL (2025-11-21 12:20:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.678944129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X........] duration: 5.808548061s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:19:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.80857944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XX....................................X.........X.X......XX.] duration: 4.878728859s trace: https://jaeger.dev.storj.io/trace/1b1ff0c2189283a2 FAIL (2025-11-21 12:19:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.878748896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X.......] duration: 5.896719647s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:19:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.896750697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X......] duration: 6.056715666s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:19:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.056746804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XXX....................................X.........X.X......XX] duration: 5.001756047s trace: https://jaeger.dev.storj.io/trace/1b1ff0c2189283a2 FAIL (2025-11-21 12:19:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001786033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X.....] duration: 5.738957796s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:18:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.739004615s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XXXX....................................X.........X.X......X] duration: 5.001356765s trace: https://jaeger.dev.storj.io/trace/1ffb995a4f5bc6bb FAIL (2025-11-21 12:18:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00138075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X....] duration: 5.87250071s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:18:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.872531968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X...] duration: 5.772146586s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:17:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.772165854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XXXXX....................................X.........X.X......] duration: 4.996437608s trace: https://jaeger.dev.storj.io/trace/14b446c06bd8592b FAIL (2025-11-21 12:17:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.99646574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X..] duration: 5.857635847s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.857667687s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [XXXXXX....................................X.........X.X.....] duration: 3.390654565s trace: https://jaeger.dev.storj.io/trace/14b446c06bd8592b FAIL (2025-11-21 12:16:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.390682999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X.] duration: 5.932529664s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:16:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.932560752s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [XXXXXXX....................................X.........X.X....] duration: 3.463665979s trace: https://jaeger.dev.storj.io/trace/14b446c06bd8592b FAIL (2025-11-21 12:15:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463699252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............X] duration: 30.244821122s trace: https://jaeger.dev.storj.io/trace/6bcbca9d952b22be FAIL (2025-11-21 12:15:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 21 12:15:20.120792627 (292h11m18.672263088s) [upload=257324]: Starting upload Nov 21 12:15:20.120896865 (292h11m18.672367326s) [upload=257324]: Done starting upload Nov 21 12:15:20.121007547 (292h11m18.672477998s) [upload=257324]: Uploading segments... Nov 21 12:15:20.121190856 (292h11m18.672661317s) [upload=257324]: Got next segment. Inline: false Nov 21 12:15:20.121226233 (292h11m18.672696694s) [upload=257324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:15:20.121231553 (292h11m18.672702014s) [upload=257324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:15:20.368274811 (292h11m18.919745272s) [seg_pos={0 0}, upload=257324]: Begin upload segment... Nov 21 12:15:20.368344223 (292h11m18.919814684s) [seg_pos={0 0}, upload=257324]: Done begin upload segment. Nov 21 12:15:20.368385121 (292h11m18.919855582s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.368440286 (292h11m18.919910747s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369183321 (292h11m18.920653782s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.369389995 (292h11m18.920860456s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.370666925 (292h11m18.922137397s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371290884 (292h11m18.922761335s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.371600002 (292h11m18.923070463s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.373126159 (292h11m18.92459665s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.374091697 (292h11m18.925562159s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.375236277 (292h11m18.926706728s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376414040 (292h11m18.927884501s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.376555870 (292h11m18.928026341s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377534573 (292h11m18.929005034s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.377875443 (292h11m18.929345904s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378380925 (292h11m18.929851386s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:20.378826844 (292h11m18.930297305s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Uploading piece... Nov 21 12:15:22.238595528 (292h11m20.790065979s) [upload=257324]: Next returned nil segment Nov 21 12:15:22.238614013 (292h11m20.790084474s) [upload=257324]: Waiting for error group managing segments... Nov 21 12:15:22.584732289 (292h11m21.13620275s) [noise=true , address=45.91.116.13:4154 , piece_id=PNRCT3TGGTLBH6ICOFZRP2A2EWNE237XTLPGNCCDQVZMC443NEOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.158555988 (292h11m22.710026449s) [noise=true , address=145.79.227.27:4690 , piece_id=YC73NGUEP7R5XGWFK6ET4KJXYCHY7ERZXY2E32B4R7E3PWBYF6CQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.212598306 (292h11m22.764068767s) [noise=true , address=145.79.227.16:4911 , piece_id=LKQ3523VB7Y5BKMY6RPQDUFDJBDS262RHHNLZTBJII2MQVWST7YQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.797571205 (292h11m23.349041666s) [noise=true , address=145.79.227.30:4762 , piece_id=IOFJSYHKYUZJETGOONOQXEMV6R5UPLEZCVK6LVUVYQU57AVW3JNA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.914915350 (292h11m23.466385811s) [noise=true , address=145.79.227.26:4655 , piece_id=Z5A73DDJ4WXEHWW72BAU7UEARPHUYRHUJFKFK3BXUV77Z27BTPWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:24.928065330 (292h11m23.479535791s) [noise=true , address=145.79.227.26:4654 , piece_id=YERXKX2NZF432NAEZ6O62S7XHNTF4KJC5Z5SUEKPAWFCVEH7U4WQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.172566968 (292h11m23.724037429s) [noise=true , address=145.79.227.24:4583 , piece_id=WUKTDDYKQVZCJW5X5LT2U626VKMSP7KQ5IU3LMBUMGWLSBYBXX5Q, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:25.605508351 (292h11m24.156978812s) [noise=true , address=145.79.227.28:4725 , piece_id=QSVWLUWNUTWEWJVEWPABED3323U7Z4TIHYFWJURUIL26A5RUZE3A, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:26.374742709 (292h11m24.92621317s) [noise=true , address=145.79.227.9:4367 , piece_id=AYUXLLA2UM5T3T6MKNX2J3WWZUP4OOFDLH5OSKRADWGOM5DQOCVQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:27.214873699 (292h11m25.76634416s) [noise=true , address=145.79.227.15:4875 , piece_id=FW6O73SX5P73DYGQ3WN3ILSATVEYG4OLOFZOQGYRVXKATDX2CYFQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:32.323443428 (292h11m30.874913889s) [noise=true , address=145.79.227.15:4873 , piece_id=HABUS2L6PP4PR3T23GMGGQHTDJ3TFALUKNG6QFDRG2ADES6QLAWA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: <nil> Nov 21 12:15:49.426368848 (292h11m47.977839309s) [noise=true , address=145.79.227.23:4545 , piece_id=K3KCP45RAJMGIM4YBPT4MBYLPLHM4JCZDPYJJ3GMF4W3RKOZB3DQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58330->145.79.227.23:4545: use of closed network connection Nov 21 12:15:49.426387544 (292h11m47.977858005s) [noise=true , address=145.79.227.11:4730 , piece_id=7YUSMBDYIZLFTNW3PYCWSWRIKBFDFNDXEQQSIACOBPGEUNPSQWMA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.11:4730: use of closed network connection Nov 21 12:15:49.426571283 (292h11m47.978041744s) [noise=true , address=145.79.227.25:4618 , piece_id=GVVMF2UI2KJDDNMJNG4M4M53HSWVNTK2SUMGLZ2ICOZGEHDTVPVA, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:45730->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426718834 (292h11m47.978189295s) [noise=true , address=145.79.227.17:4971 , piece_id=ZBGWSBQDY7KXVSMZXBYAULYM467RECQPHUSQSW4SPI3TIMC4UKOQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33380->145.79.227.17:4971: use of closed network connection Nov 21 12:15:49.426754642 (292h11m47.978225103s) [noise=true , address=145.79.227.20:4439 , piece_id=7UYL3HPKR43YQPWLXXZIJE4K3ECNIZYGCGVGMBWH52MCQ6XCVBYQ, seg_pos={0 0}, upload=257324]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 188.166.225.170:54156->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:15:49.426793957 (292h11m47.978264418s) [seg_pos={0 0}, upload=257324]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 21 12:15:49.668413768 (292h11m48.219884239s) [upload=257324]: Done uploading segments... Nov 21 12:15:49.668460747 (292h11m48.219931208s) [upload=257324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.244848373s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XXXXXXXX....................................X.........X.X...] duration: 4.963769796s trace: https://jaeger.dev.storj.io/trace/14b446c06bd8592b FAIL (2025-11-21 12:15:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.96379884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XXXXXXXXX....................................X.........X.X..] duration: 3.440090767s trace: https://jaeger.dev.storj.io/trace/14b446c06bd8592b FAIL (2025-11-21 12:14:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440121635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XXXXXXXXXX....................................X.........X.X.] duration: 5.001663851s trace: https://jaeger.dev.storj.io/trace/14b446c06bd8592b FAIL (2025-11-21 12:14:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001696103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.............] duration: 5.917733188s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:14:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.917751212s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XXXXXXXXXXX....................................X.........X.X] duration: 5.001622601s trace: https://jaeger.dev.storj.io/trace/14b446c06bd8592b FAIL (2025-11-21 12:13:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001648891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......X....X.X.X..X.X..X...X..XXX.X.XX...X...XX............] duration: 5.782621373s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:13:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.782652792s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XXXXXXXXXXXX....................................X.........X.] duration: 3.687437873s trace: https://jaeger.dev.storj.io/trace/59768f4f2afc0904 FAIL (2025-11-21 12:13:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.687460348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........X....X.X.X..X.X..X...X..XXX.X.XX...X...XX...........] duration: 5.952982691s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:13:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.953014112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X....X.X.X..X.X..X...X..XXX.X.XX...X...XX..........] duration: 5.807431635s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.807455962s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX....................................X.........X] duration: 5.001212072s trace: https://jaeger.dev.storj.io/trace/59768f4f2afc0904 FAIL (2025-11-21 12:12:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001228272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.........] duration: 5.860182243s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:11:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.860239392s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX....................................X.........] duration: 3.716276147s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:11:45 UTC) orbiter warning: context deadline exceeded Duration: 3.716328727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........X....X.X.X..X.X..X...X..XXX.X.XX...X...XX........] duration: 5.771538907s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:11:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.771572222s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX....................................X........] duration: 3.760448592s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:11:20 UTC) orbiter warning: context deadline exceeded Duration: 3.760478087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.......] duration: 5.934586302s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:11:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.93464242s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)