Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 380.194539ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:23:56 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 380.225979ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 5.979810203s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:23:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.979858705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 6.012680736s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:22:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.012718688s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 46.342715ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:22:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 46.3762ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 5.971035203s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:22:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.971057635s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 45.592487ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:22:05 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 45.643233ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 6.00829805s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:21:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.008370719s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [X...X..........] duration: 50.429799ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:21:47 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.457603ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 330.82751ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:21:26 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 330.977074ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.X...X.........] duration: 77.815335ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:21:02 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.917431ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 6.080387098s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:20:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.080417838s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 343.15858ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:20:39 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 343.202665ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [..X...X........] duration: 52.78927ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:20:22 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.817194ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 6.013355159s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:20:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.013456803s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 376.207141ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:20:01 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 376.241417ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 6.127303489s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:19:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.127331912s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [...X...X.......] duration: 52.890905ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:19:21 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.974604ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.974100766s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:19:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.974131914s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 314.522553ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:18:49 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 314.549175ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.917420399s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:18:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.917439134s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [....X...X......] duration: 59.420087ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:18:35 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 59.45258ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.786863644s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:18:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.786897228s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 312.798488ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:18:01 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 312.93539ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.....X...X.....] duration: 64.182816ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:17:58 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.20022ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......................................XX..................] duration: 6.016886232s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:17:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.016918173s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [......X...X....] duration: 51.340833ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:17:25 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 51.368666ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 334.558335ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:17:20 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 334.585736ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.897559254s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:17:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.897591956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......................................XX.................] duration: 5.971345087s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:16:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.971528417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.768743371s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:16:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.768774472s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.......X...X...] duration: 53.444919ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:16:34 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.473414ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......................................XX................] duration: 6.010217347s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:16:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.010262483s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 330.171761ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:16:14 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 330.195355ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [........X...X..] duration: 48.884595ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:16:11 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 48.913371ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......................................XX...............] duration: 5.996014988s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:15:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.996043243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.692072453s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:15:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.692107209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.696338287s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:15:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.696367312s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 622.400983ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:15:12 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 622.434026ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.........X...X.] duration: 85.966817ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:15:08 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 85.985754ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.028887254s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:15:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.028920157s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [..........X...X] duration: 219.344674ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6e45b1426d24f455 FAIL (2025-11-14 02:14:44 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:44732->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 219.371184ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 324.669394ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:14:43 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 324.695945ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......................................XX..............] duration: 5.890201553s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:14:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.890231379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.615142625s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:14:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.615171681s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 313.112467ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:14:20 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 313.139899ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.788003105s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:14:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.788025847s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 326.810982ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/72106348ae9e9138 FAIL (2025-11-14 02:13:59 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33230->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 326.83611ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......................................XX.............] duration: 5.852830361s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:13:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.852862512s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 53.079032ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3425d218113d21b1 FAIL (2025-11-14 02:13:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59112->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.115603ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.72394253s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 02:13:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.723971345s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)