Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 52.682689ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:39:54 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.742763ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 6.034733811s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.034844904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 6.04442875s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:39:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.04445903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.754297017s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:39:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.754335791s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 75.976796ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:39:19 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 75.9999ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.87770437s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:38:57 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.877738114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 6.045189055s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:38:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.045218461s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 75.666224ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:38:21 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 75.686504ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.813837927s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:37:57 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.813865692s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 51.409354ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:37:31 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 51.440964ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 5.926161207s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:37:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.926191253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.820276746s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:37:16 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.820306423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.748140134s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:36:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.748188305s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 372.107639ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:36:44 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 372.138117ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 5.964706954s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:36:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.964758231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.752649002s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:35:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.752678056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 5.928449612s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:35:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.928480931s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 361.036438ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:35:46 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 361.063299ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.83990294s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:35:21 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.839934271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 5.941835603s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:35:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.941885518s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 50.477641ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:34:52 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.507176ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 5.955292686s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:34:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.95532602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.906313309s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:34:21 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.906385417s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 61.873178ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:34:11 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 61.897656ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 53.230302ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:33:33 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.262242ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.77231478s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:33:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.772340669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 6.024771891s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:33:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.024842285s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 47.56802ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:32:46 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 47.613776ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 6.019219443s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:32:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.019250271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.805566294s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:32:18 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.805596182s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 55.262902ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:32:17 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 55.288531ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 66.831961ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:31:53 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 66.860666ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 6.758995695s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:31:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.759028656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.878665432s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:31:17 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.878695329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 6.099929207s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:31:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.099957831s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 76.279573ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:30:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 76.305733ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 56.301971ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:30:26 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.330375ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 6.187301952s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:30:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.18733719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.81384277s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:30:19 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.813878147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 5.965114539s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:29:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.965136361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.894328647s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:29:30 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.894359145s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 246.791612ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4e89a8009dea74d9 FAIL (2025-11-14 02:29:22 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57014->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 246.816911ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 5.965002308s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:28:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.965030882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.820390765s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:28:45 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.820418218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 5.981246524s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:27:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.981274247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.795882681s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:27:35 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.795909311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 5.963771078s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:26:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.963796006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.707061933s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:26:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.707087081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.921525989s trace: https://jaeger.dev.storj.io/trace/4d67fad9e2c6ec76 FAIL (2025-11-14 02:26:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.881728976 (114h21m4.433199447s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882337685 (114h21m4.433808146s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.882761321 (114h21m4.434231782s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883755024 (114h21m4.435225485s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.883875503 (114h21m4.435345974s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.884804482 (114h21m4.436274943s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885204404 (114h21m4.436674865s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.885789247 (114h21m4.437259708s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886160905 (114h21m4.437631376s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.886974594 (114h21m4.438445055s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887182179 (114h21m4.438652651s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.887657485 (114h21m4.439127946s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888130356 (114h21m4.439600817s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:05.888691123 (114h21m4.440161594s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Uploading piece... Nov 14 02:25:07.255882067 (114h21m5.807352528s) [upload=100627]: Next returned nil segment Nov 14 02:25:07.255900562 (114h21m5.807371023s) [upload=100627]: Waiting for error group managing segments... Nov 14 02:25:07.629575566 (114h21m6.181046027s) [noise=true , address=64.49.13.155:4914 , piece_id=OS3G6HXXU2KG3YLW4U2VJHC4FIN6I24LXBH2PPS77SYKAKDDJCAA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.802313484 (114h21m6.353783945s) [noise=true , address=64.49.13.140:4510 , piece_id=WRZE6WU5UZ56L22NNBV3GVUG3HC52W7AJFP6ULZSLQ25V76JEEQQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.903131683 (114h21m6.454602145s) [noise=true , address=77.90.31.19:4865 , piece_id=EATYIM7P24ULDH5KNKIJMDBBM2JIWA5FELUV66EB2G2N7LTHQEGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.907810763 (114h21m6.459281224s) [noise=true , address=45.91.116.9:4269 , piece_id=4XXWWRRNH6T2DZS5EKHU6W2DASHNOWT6DSD4WZHLHFNNGQ7WGZSA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.915612680 (114h21m6.467083141s) [noise=true , address=45.91.116.11:4128 , piece_id=R46ZRM4NJ6ENKDIVKNMGMOIO4DJUXPYBLLZLY3JKL2BZ3IMA5CVA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.919665296 (114h21m6.471135757s) [noise=true , address=77.90.31.13:4654 , piece_id=QKBPHJI4SVI4NIA2QRTIKIMN35NLTC56RQK7RSKDUAHUMSJWB3ZA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.927705556 (114h21m6.479176018s) [noise=true , address=77.90.31.14:4669 , piece_id=NSWMZUFVUUWQI75Q46ON6AT7MYIIIASCPXE6EVUEEZVLLFV2TRMQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937100136 (114h21m6.488570608s) [noise=true , address=77.90.31.10:4523 , piece_id=XQJOQWJZV5MS3HYYQCXNYTDDH356CLYA4W6QDFQJEOTZ3GCB2TFA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.937995852 (114h21m6.489466323s) [noise=true , address=45.91.116.12:4243 , piece_id=HQPJKQANYLCNVOT4QEZ5JQY7YS6SE2675M3GD2RPMXD3UGX22E5A, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.938767291 (114h21m6.490237752s) [noise=true , address=77.90.31.18:4809 , piece_id=FS5BSPEWP734H6KZQ3ILIC7BU532CJ2NHM2K3XL3G5CTM5G26TOA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:07.975450604 (114h21m6.526921065s) [noise=true , address=45.91.116.6:4000 , piece_id=CDMDGQY3XQT7FGULOUMCBUEKLM3PWX5BA7SXBMKOMA6JH7KCI3YQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:08.060141452 (114h21m6.611611913s) [noise=true , address=77.90.31.11:4572 , piece_id=LNPRUL5YLDXS3KXUGO72RVPXYHV43Y3PMZZ4LIE3WX2SQW6VHRNA, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:25:09.481028605 (114h21m8.032499066s) [noise=true , address=5.231.73.5:4016 , piece_id=TNMME4VZUSYBQPEDBOT6EWOSRO4CWM3NWKX6YBLVMLDKYCLJYZAQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364334249 (114h22m2.91580471s) [noise=true , address=145.79.227.20:4442 , piece_id=M5FENIZ665TWULGDISFLB5UQMBI7QZUAFRCHZUE7776IX7QDDRGQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: <nil> Nov 14 02:26:04.364362523 (114h22m2.915832984s) [seg_pos={0 0}, upload=100627]: Segment reached optimal threshold of 14 pieces. Nov 14 02:26:04.364726596 (114h22m2.916197057s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56520->145.79.227.33:4872: use of closed network connection Nov 14 02:26:04.364736675 (114h22m2.916207136s) [noise=true , address=145.79.227.33:4872 , piece_id=YCRQ5S27UDJH5VJUBYIGJLXGPJ7B22MTFMAAAHKB25PEIEGULPDQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364900176 (114h22m2.916370647s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BZnnbpp63tb9YBYEym3fuvJB4fR1L2GPTpWPbESbVgAXHpa64): protocol: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->145.79.227.50:4395: use of closed network connection Nov 14 02:26:04.364912240 (114h22m2.916382711s) [noise=true , address=145.79.227.50:4395 , piece_id=OIRL6AAITMD75BYL5TY6VECW2FBVTGKEKB447AOSOGIC454TQLEQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.364936345 (114h22m2.916406806s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YdSTc2X61baRotZGev4q25BFkEspbienJtMcpbQvGgbyCRYfx): protocol: read tcp 188.166.225.170:48200->145.79.227.37:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 02:26:04.364944490 (114h22m2.916414941s) [noise=true , address=145.79.227.37:4586 , piece_id=E5OULRJEXSY7JAJJMOO4DMZ6SPJ5TTGSU7RUV77QKRAVG3ACQF5Q, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365027369 (114h22m2.91649783s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv): protocol: expected piece hash; context canceled; EOF Nov 14 02:26:04.365037989 (114h22m2.91650845s) [noise=true , address=5.231.73.3:4036 , piece_id=NFQPRUM6KAY7SZ76QNPRPGPOMA5KE65MPJUOAT7J7V5YEHYGWAXQ, seg_pos={0 0}, upload=100627]: Long tail cancellation due to optimal threshold Nov 14 02:26:04.365060712 (114h22m2.916531173s) [seg_pos={0 0}, upload=100627]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 02:26:04.365180710 (114h22m2.916651172s) [upload=100627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 02:26:04.365188695 (114h22m2.916659156s) [upload=100627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 02:26:04.675897038 (114h22m3.227367509s) [upload=100627]: Done uploading segments... Nov 14 02:26:04.675924961 (114h22m3.227395423s) [upload=100627]: Upload finished. err: <nil> Duration: 5.921542841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 6.024471178s trace: https://jaeger.dev.storj.io/trace/71a2560a3cd5f05f FAIL (2025-11-14 02:26:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ov 14 02:04:46.378275843 (114h0m44.929746304s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378305720 (114h0m44.929776191s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.378799600 (114h0m44.930270051s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379218017 (114h0m44.930688478s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.379835743 (114h0m44.931306194s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380410026 (114h0m44.931880477s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.380813515 (114h0m44.932283976s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.381669986 (114h0m44.933140447s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.382281049 (114h0m44.93375151s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.383296683 (114h0m44.934767134s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.384780860 (114h0m44.936251321s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.387428271 (114h0m44.938898722s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.389148706 (114h0m44.940619167s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.391724832 (114h0m44.943195293s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.392615177 (114h0m44.944085638s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:46.394587103 (114h0m44.946057564s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Uploading piece... Nov 14 02:04:48.138276742 (114h0m46.689747193s) [upload=100326]: Next returned nil segment Nov 14 02:04:48.138292292 (114h0m46.689762753s) [upload=100326]: Waiting for error group managing segments... Nov 14 02:04:48.474991600 (114h0m47.026462061s) [noise=true , address=45.91.116.8:4045 , piece_id=NWZNM4GNKXJBW46AVVFOIRDVR6HY5EHSXZKOR2VECTDJE5IXHA5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.500285966 (114h0m47.051756427s) [noise=true , address=77.90.31.17:4772 , piece_id=4EHDCCKSULNHNI7BK7JLA7BI7VCCW3IXBCGC6FDOCZPBJ2W7W4IA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.520373816 (114h0m47.071844278s) [noise=true , address=77.90.31.14:4661 , piece_id=W575JJKJ6JETJ75Y26XIAUR5RZZMRSAIUFVULWTUMKRDT4BCUELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.532987937 (114h0m47.084458398s) [noise=true , address=45.91.116.14:4189 , piece_id=R7K3X7RT6OHD5X452YPM7A7XYECOSRR26W5JVDJMMTO6RTFKXAFQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.548122695 (114h0m47.099593156s) [noise=true , address=77.90.31.15:4699 , piece_id=3C5REUC4FGIDJQXTYSXRZVPTKEIBLJS3DEKDVRM66LD4CGQ2XELQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.575823303 (114h0m47.127293754s) [noise=true , address=45.91.116.10:4081 , piece_id=V5ORSKW3TB3SDIQ3VBN7MHANWGZFBNWAI5CMUSMCCYE7F3WGUZHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.646757344 (114h0m47.198227805s) [noise=true , address=45.91.116.9:4262 , piece_id=FFZ6C6U63Z4GS6P4L7UQYLADCBDFJXT5ZKPJIREZQVGR4FC7YYQQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:04:48.703779705 (114h0m47.255250166s) [noise=true , address=77.90.31.11:4555 , piece_id=FVQYK2F5QXYXYDPPIDK6WK5Y5TTPDRSXOEAKPEWFM25FIBEIYIPA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:11.168570002 (114h1m9.720040463s) [noise=true , address=145.79.227.9:4366 , piece_id=TWG7RJVPGUCWSESCCF6WMARWO5PZR3EUHQSD4Y4FCRMJSEZ42TMA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: <nil> Nov 14 02:05:15.414272193 (114h1m13.965742654s) [noise=true , address=145.79.227.30:4761 , piece_id=763FNQ2PU7DONYSBGC5QU6HTZWHK3UU66BNLUTTGWQYXWZNUMU6Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54058->145.79.227.30:4761: use of closed network connection Nov 14 02:05:15.414443269 (114h1m13.96591373s) [noise=true , address=145.79.227.21:4473 , piece_id=63MPGV62KTVCUBKQACOA43HITANDKRH7HFGRPRGMHCXSY5AX4TSA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.21:4473: use of closed network connection Nov 14 02:05:15.414633501 (114h1m13.966103962s) [noise=true , address=145.79.227.27:4690 , piece_id=QUMKENVN3WFFK7B6MLCNFXQ3LVGZIDY7F2EIF5UGLAUTOWNDBZDQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->145.79.227.27:4690: use of closed network connection Nov 14 02:05:15.414777275 (114h1m13.966247736s) [noise=true , address=145.79.227.22:4509 , piece_id=6UGKUPCOYZFFDWFGM4Z53FRDETRSS35BB3DCUFZUYOFMZBIG4GHQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47996->145.79.227.22:4509: use of closed network connection Nov 14 02:05:15.414954332 (114h1m13.966424793s) [noise=true , address=145.79.227.31:4798 , piece_id=DQAH7NCAOCVJHHNHZXHJH6JOZI5AE4524JYOCQ4IS5624VOC54XA, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->145.79.227.31:4798: use of closed network connection Nov 14 02:05:15.415103035 (114h1m13.966573507s) [noise=true , address=145.79.227.28:4727 , piece_id=ZFDHQZPZS6Q5BHQQZHQDZ3LUDWPQUUFWD3BZ4ZUH5MX2DAWBZZUQ, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41674->145.79.227.28:4727: use of closed network connection Nov 14 02:05:15.415481074 (114h1m13.966951536s) [noise=true , address=145.79.227.11:4730 , piece_id=EW6OCFHJY7N3AGZ7O3WSO73XEP55N4SVM7RO32Z3CKCJA56NAL5Q, seg_pos={0 0}, upload=100326]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 02:05:15.415530489 (114h1m13.96700095s) [seg_pos={0 0}, upload=100326]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 14 02:05:15.670119394 (114h1m14.221589865s) [upload=100326]: Done uploading segments... Nov 14 02:05:15.670167336 (114h1m14.221637797s) [upload=100326]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.024500443s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)