Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.753847419s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:59:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.753881393s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X.........X...............................X...X] duration: 5.00213671s trace: https://jaeger.dev.storj.io/trace/444bf3f58671b75 FAIL (2025-11-11 17:59:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002190753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.286945145s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:58:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.28696396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.752472842s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:58:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.752506978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X.........X...............................X...] duration: 3.971637583s trace: https://jaeger.dev.storj.io/trace/4d41f160f9064ac3 FAIL (2025-11-11 17:57:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.971665016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.725661061s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:57:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.725688522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.105836294s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:57:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.105869185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X.........X...............................X..] duration: 3.943972553s trace: https://jaeger.dev.storj.io/trace/4d41f160f9064ac3 FAIL (2025-11-11 17:57:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.944005145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X.........X...............................X.] duration: 3.920992252s trace: https://jaeger.dev.storj.io/trace/4d41f160f9064ac3 FAIL (2025-11-11 17:56:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.921024122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.25847378s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:56:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.258534695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 7.479300406s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:56:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.479332337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.679887698s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:56:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.679925838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X.........X...............................X] duration: 5.000996882s trace: https://jaeger.dev.storj.io/trace/4d41f160f9064ac3 FAIL (2025-11-11 17:55:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001039723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.881573314s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:55:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.881600675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X.........X...............................] duration: 3.933746879s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:55:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.933847841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.72348s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:55:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.723514607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.994124477s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:54:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.994155336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X.........X..............................] duration: 3.958566963s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:54:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.95876487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.629589093s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:54:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.629743327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.06990121s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:54:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.069931467s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 48.838639ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:54:21 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 48.865732ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X.........X.............................] duration: 3.952428114s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:54:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.952463522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.450655245s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:53:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.450685121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................X.......................................] duration: 6.619751571s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:53:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.619782431s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X.........X............................] duration: 4.131493996s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:53:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.131519363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.167553494s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.167579384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX...................X......................................] duration: 7.148576903s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:53:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.148619553s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 50.501786ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:53:16 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.529089ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X.........X...........................] duration: 3.980861218s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:53:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.980891846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.258743651s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:52:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.258764872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XX...................X.....................................] duration: 7.980055765s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:52:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.980100371s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X.........X..........................] duration: 3.959266536s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:52:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.959337933s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 65.817782ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:52:31 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 65.849643ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.274792241s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:52:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.274824623s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 53.601249ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:52:06 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.62293ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.275742753s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:51:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.275776759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XX...................X....................................] duration: 7.861767193s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:51:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.861797981s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 47.202574ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:51:31 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 47.229637ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X.........X.........................] duration: 4.061185951s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:51:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.061215048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.699801665s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:51:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.69983643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.381215488s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:50:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.381273397s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 51.731678ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:50:30 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 51.830817ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XX...................X...................................] duration: 6.987138677s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:50:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.987172441s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X.........X........................] duration: 4.05836059s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:50:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.058392842s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 55.999757ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:50:06 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.030695ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.19309546s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:50:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.193130705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XX...................X..................................] duration: 6.493139949s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:49:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.49316605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XX...................X.................................] duration: 6.806216611s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:49:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.806255435s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X.........X.......................] duration: 4.148004745s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:49:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.148038979s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 57.847335ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:49:15 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 57.876128ms core website

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)